"Chunked" MemoryStream

Question

Je cherche l'implémentation de MemoryStream qui n'alloue pas la mémoire comme un gros bloc, mais plutôt comme une collection de morceaux. Je veux stocker quelques Go de données en mémoire (64 bits) et éviter la limitation de la fragmentation de la mémoire.

Answer 1

Vous devez d'abord déterminer si la fragmentation d'adresse virtuelle est le problème.

Si vous êtes sur une machine 64 bits (que vous semblez indiquer que vous êtes), j'en doute sérieusement. Chaque processus 64 bits a presque tout l'espace de mémoire virtuelle 64 bits disponible et votre seul souci est la fragmentation de l'espace d'adressage virtuel et non la fragmentation de la mémoire physique (ce qui est le souci du système d'exploitation). Le gestionnaire de mémoire du système d'exploitation enregistre déjà la mémoire sous les couvertures. Dans un avenir prévisible, vous ne manquerez plus d'espace d'adressage virtuel avant d'avoir épuisé votre mémoire physique. Ce changement est peu probable avant que nous prenions notre retraite.

Si vous disposez d'un espace d'adressage de 32 bits, puis d'allouer de gros blocs de mémoire contigus dans la branche GB, vous rencontrerez un problème de fragmentation assez rapidement. Il n'y a pas de flux de mémoire allouant un bloc de mémoire dans le CLR. Il y en a un sous la couverture dans ASP.NET (pour d'autres raisons) mais il n'est pas accessible. Si vous devez emprunter ce chemin, il est probablement préférable d'en écrire vous-même car le modèle d'utilisation de votre application est peu susceptible d'être similaire à celui de beaucoup d'autres et essayer d'adapter vos données à un espace d'adressage 32 bits.

Je recommande fortement d'avoir besoin d'un processus 64 bits si vous manipulez des Go de données. Il fera un bien meilleur travail que les solutions roulées à la main pour la fragmentation de l'espace d'adressage 32 bits, peu importe la façon dont vous êtes cleaver.

Answer 2

Quelque chose comme ceci:

class ChunkedMemoryStream : Stream 
{ 
    private readonly List<byte[]> _chunks = new List<byte[]>(); 
    private int _positionChunk; 
    private int _positionOffset; 
    private long _position; 

    public override bool CanRead 
    { 
     get { return true; } 
    } 

    public override bool CanSeek 
    { 
     get { return true; } 
    } 

    public override bool CanWrite 
    { 
     get { return true; } 
    } 

    public override void Flush() { } 

    public override long Length 
    { 
     get { return _chunks.Sum(c => c.Length); } 
    } 

    public override long Position 
    { 
     get 
     { 
      return _position; 
     } 
     set 
     { 
      _position = value; 

      _positionChunk = 0; 

      while (_positionOffset != 0) 
      { 
       if (_positionChunk >= _chunks.Count) 
        throw new OverflowException(); 

       if (_positionOffset < _chunks[_positionChunk].Length) 
        return; 

       _positionOffset -= _chunks[_positionChunk].Length; 
       _positionChunk++; 
      } 
     } 
    } 

    public override int Read(byte[] buffer, int offset, int count) 
    { 
     int result = 0; 
     while ((count != 0) && (_positionChunk != _chunks.Count)) 
     { 
      int fromChunk = Math.Min(count, _chunks[_positionChunk].Length - _positionOffset); 
      if (fromChunk != 0) 
      { 
       Array.Copy(_chunks[_positionChunk], _positionOffset, buffer, offset, fromChunk); 
       offset += fromChunk; 
       count -= fromChunk; 
       result += fromChunk; 
       _position += fromChunk; 
      } 

      _positionOffset = 0; 
      _positionChunk++; 
     } 
     return result; 
    } 

    public override long Seek(long offset, SeekOrigin origin) 
    { 
     long newPos = 0; 

     switch (origin) 
     { 
      case SeekOrigin.Begin: 
       newPos = offset; 
       break; 
      case SeekOrigin.Current: 
       newPos = Position + offset; 
       break; 
      case SeekOrigin.End: 
       newPos = Length - offset; 
       break; 
     } 

     Position = Math.Max(0, Math.Min(newPos, Length)); 
     return newPos; 
    } 

    public override void SetLength(long value) 
    { 
     throw new NotImplementedException(); 
    } 

    public override void Write(byte[] buffer, int offset, int count) 
    { 
     while ((count != 0) && (_positionChunk != _chunks.Count)) 
     { 
      int toChunk = Math.Min(count, _chunks[_positionChunk].Length - _positionOffset); 
      if (toChunk != 0) 
      { 
       Array.Copy(buffer, offset, _chunks[_positionChunk], _positionOffset, toChunk); 
       offset += toChunk; 
       count -= toChunk; 
       _position += toChunk; 
      } 

      _positionOffset = 0; 
      _positionChunk++; 
     } 

     if (count != 0) 
     { 
      byte[] chunk = new byte[count]; 
      Array.Copy(buffer, offset, chunk, 0, count); 
      _chunks.Add(chunk); 
      _positionChunk = _chunks.Count; 
      _position += count; 
     } 
    } 
} 

class Program 
{ 
    static void Main(string[] args) 
    { 
     ChunkedMemoryStream cms = new ChunkedMemoryStream(); 

     Debug.Assert(cms.Length == 0); 
     Debug.Assert(cms.Position == 0); 

     cms.Position = 0; 

     byte[] helloworld = Encoding.UTF8.GetBytes("hello world"); 

     cms.Write(helloworld, 0, 3); 
     cms.Write(helloworld, 3, 3); 
     cms.Write(helloworld, 6, 5); 

     Debug.Assert(cms.Length == 11); 
     Debug.Assert(cms.Position == 11); 

     cms.Position = 0; 

     byte[] b = new byte[20]; 
     cms.Read(b, 3, (int)cms.Length); 
     Debug.Assert(b.Skip(3).Take(11).SequenceEqual(helloworld)); 

     cms.Position = 0; 
     cms.Write(Encoding.UTF8.GetBytes("seeya"), 0, 5); 

     Debug.Assert(cms.Length == 11); 
     Debug.Assert(cms.Position == 5); 

     cms.Position = 0; 
     cms.Read(b, 0, (byte) cms.Length); 
     Debug.Assert(b.Take(11).SequenceEqual(Encoding.UTF8.GetBytes("seeya world"))); 

     Debug.Assert(cms.Length == 11); 
     Debug.Assert(cms.Position == 11); 

     cms.Write(Encoding.UTF8.GetBytes(" again"), 0, 6); 

     Debug.Assert(cms.Length == 17); 
     Debug.Assert(cms.Position == 17); 

     cms.Position = 0; 
     cms.Read(b, 0, (byte)cms.Length); 
     Debug.Assert(b.Take(17).SequenceEqual(Encoding.UTF8.GetBytes("seeya world again"))); 

    } 
} 

Answer 3

vous pouvez utiliser ce

http://www.123aspx.com/Rotor/src/chunkedmemorystream.cs.txt

Answer 4

J'ai trouvé le même problème dans ma demande. J'ai lu une grande quantité de données compressées et j'ai souffert de OutOfMemoryException en utilisant MemoryStream. J'ai écrit ma propre implémentation de flux de mémoire "chunked" basé sur la collection de tableaux d'octets. Si vous avez une idée de comment rendre ce flux de mémoire plus efficace, s'il vous plaît écrivez-moi à ce sujet.

public sealed class ChunkedMemoryStream : Stream 
{ 
    #region Constants 

    private const int BUFFER_LENGTH = 65536; 
    private const byte ONE = 1; 
    private const byte ZERO = 0; 

    #endregion 

    #region Readonly & Static Fields 

    private readonly Collection<byte[]> _chunks; 

    #endregion 

    #region Fields 

    private long _length; 

    private long _position; 
    private const byte TWO = 2; 

    #endregion 

    #region C'tors 

    public ChunkedMemoryStream() 
    { 
     _chunks = new Collection<byte[]> { new byte[BUFFER_LENGTH], new byte[BUFFER_LENGTH] }; 
     _position = ZERO; 
     _length = ZERO; 
    } 

    #endregion 

    #region Instance Properties 

    public override bool CanRead 
    { 
     get { return true; } 
    } 

    public override bool CanSeek 
    { 
     get { return true; } 
    } 

    public override bool CanWrite 
    { 
     get { return true; } 
    } 

    public override long Length 
    { 
     get { return _length; } 
    } 

    public override long Position 
    { 
     get { return _position; } 
     set 
     { 
      if (!CanSeek) 
       throw new NotSupportedException(); 

      _position = value; 

      if (_position > _length) 
       _position = _length - ONE; 
     } 
    } 


    private byte[] CurrentChunk 
    { 
     get 
     { 
      long positionDividedByBufferLength = _position/BUFFER_LENGTH; 
      var chunkIndex = Convert.ToInt32(positionDividedByBufferLength); 
      byte[] chunk = _chunks[chunkIndex]; 
      return chunk; 
     } 
    } 

    private int PositionInChunk 
    { 
     get 
     { 
      int positionInChunk = Convert.ToInt32(_position % BUFFER_LENGTH); 
      return positionInChunk; 
     } 
    } 

    private int RemainingBytesInCurrentChunk 
    { 
     get 
     { 
      Contract.Ensures(Contract.Result<int>() > ZERO); 
      int remainingBytesInCurrentChunk = CurrentChunk.Length - PositionInChunk; 
      return remainingBytesInCurrentChunk; 
     } 
    } 

    #endregion 

    #region Instance Methods 

    public override void Flush() 
    { 
    } 

    public override int Read(byte[] buffer, int offset, int count) 
    { 
     if (offset + count > buffer.Length) 
      throw new ArgumentException(); 

     if (buffer == null) 
      throw new ArgumentNullException(); 

     if (offset < ZERO || count < ZERO) 
      throw new ArgumentOutOfRangeException(); 

     if (!CanRead) 
      throw new NotSupportedException(); 

     int bytesToRead = count; 
     if (_length - _position < bytesToRead) 
      bytesToRead = Convert.ToInt32(_length - _position); 

     int bytesreaded = 0; 
     while (bytesToRead > ZERO) 
     { 
      // get remaining bytes in current chunk 
      // read bytes in current chunk 
      // advance to next position 
      int remainingBytesInCurrentChunk = RemainingBytesInCurrentChunk; 
      if (remainingBytesInCurrentChunk > bytesToRead) 
       remainingBytesInCurrentChunk = bytesToRead; 
      Array.Copy(CurrentChunk, PositionInChunk, buffer, offset, remainingBytesInCurrentChunk); 
      //move position in source 
      _position += remainingBytesInCurrentChunk; 
      //move position in target 
      offset += remainingBytesInCurrentChunk; 
      //bytesToRead is smaller 
      bytesToRead -= remainingBytesInCurrentChunk; 
      //count readed bytes; 
      bytesreaded += remainingBytesInCurrentChunk; 
     } 
     return bytesreaded; 
    } 

    public override long Seek(long offset, SeekOrigin origin) 
    { 
     switch (origin) 
     { 
      case SeekOrigin.Begin: 
       Position = offset; 
       break; 
      case SeekOrigin.Current: 
       Position += offset; 
       break; 
      case SeekOrigin.End: 
       Position = Length + offset; 
       break; 
     } 
     return Position; 
    } 

    private long Capacity 
    { 
     get 
     { 
      int numberOfChunks = _chunks.Count; 


      long capacity = numberOfChunks * BUFFER_LENGTH; 
      return capacity; 
     } 
    } 

    public override void SetLength(long value) 
    { 
     if (value > _length) 
     { 
      while (value > Capacity) 
      { 
       var item = new byte[BUFFER_LENGTH]; 
       _chunks.Add(item); 
      } 
     } 
     else if (value < _length) 
     { 
      var decimalValue = Convert.ToDecimal(value); 
      var valueToBeCompared = decimalValue % BUFFER_LENGTH == ZERO ? Capacity : Capacity - BUFFER_LENGTH; 
      //remove data chunks, but leave at least two chunks 
      while (value < valueToBeCompared && _chunks.Count > TWO) 
      { 
       byte[] lastChunk = _chunks.Last(); 
       _chunks.Remove(lastChunk); 
      } 
     } 
     _length = value; 
     if (_position > _length - ONE) 
      _position = _length == 0 ? ZERO : _length - ONE; 
    } 

    public override void Write(byte[] buffer, int offset, int count) 
    { 
     if (!CanWrite) 
      throw new NotSupportedException(); 

     int bytesToWrite = count; 

     while (bytesToWrite > ZERO) 
     { 
      //get remaining space in current chunk 
      int remainingBytesInCurrentChunk = RemainingBytesInCurrentChunk; 

      //if count of bytes to be written is fewer than remaining 
      if (remainingBytesInCurrentChunk > bytesToWrite) 
       remainingBytesInCurrentChunk = bytesToWrite; 

      //if remaining bytes is still greater than zero 
      if (remainingBytesInCurrentChunk > ZERO) 
      { 
       //write remaining bytes to current Chunk 

       Array.Copy(buffer, offset, CurrentChunk, PositionInChunk, remainingBytesInCurrentChunk); 

       //change offset of source array 
       offset += remainingBytesInCurrentChunk; 
       //change bytes to write 
       bytesToWrite -= remainingBytesInCurrentChunk; 
       //change length and position 
       _length += remainingBytesInCurrentChunk; 
       _position += remainingBytesInCurrentChunk; 
      } 

      if (Capacity == _position) 
       _chunks.Add(new byte[BUFFER_LENGTH]); 
     } 
    } 

    /// <summary> 
    ///  Gets entire content of stream regardless of Position value and return output as byte array 
    /// </summary> 
    /// <returns>byte array</returns> 
    public byte[] ToArray() 
    { 
     var outputArray = new byte[Length]; 
     if (outputArray.Length != ZERO) 
     { 
      long outputPosition = ZERO; 
      foreach (byte[] chunk in _chunks) 
      { 
       var remainingLength = (Length - outputPosition) > chunk.Length 
              ? chunk.Length 
              : Length - outputPosition; 
       Array.Copy(chunk, ZERO, outputArray, outputPosition, remainingLength); 
       outputPosition = outputPosition + remainingLength; 
      } 
     } 
     return outputArray; 
    } 

    /// <summary> 
    ///  Method set Position to first element and write entire stream to another 
    /// </summary> 
    /// <param name="stream">Target stream</param> 
    public void WriteTo(Stream stream) 
    { 
     Contract.Requires(stream != null); 

     Position = ZERO; 
     var buffer = new byte[BUFFER_LENGTH]; 
     int bytesReaded; 
     do 
     { 
      bytesReaded = Read(buffer, ZERO, BUFFER_LENGTH); 
      stream.Write(buffer, ZERO, bytesReaded); 
     } while (bytesReaded > ZERO); 
    } 

    #endregion 
} 

Answer 5

Vous devez utiliser le UnmanagedMemoryStream lorsqu'ils traitent avec plus de 2 Go de mémoire morceaux, comme MemoryStream est limitée à 2 Go, et le UnmanagedMemoryStream a été fait pour faire face à ce problème.

Answer 6

SparseMemoryStream le fait dans .NET il est cependant enfoui dans une bibliothèque de classes interne - le code source est bien sûr disponible, puisque Microsoft l'a mis à jour en open source.

Vous pouvez saisir le code ici: http://www.dotnetframework.org/default.aspx/4@0/4@0/DEVDIV_TFS/Dev10/Releases/RTMRel/wpf/src/Base/MS/Internal/IO/Packaging/SparseMemoryStream@cs/1305600/SparseMemoryStream@cs

Cela étant dit, je le recommande vivement ne l'utilise pas comme il est - à tout le moins supprimer tous les appels à IsolatedStorage pour commencer, comme cela semble être la cause de l'absence de fin de bugs * dans l'API de packaging du framework. (*: En plus de répandre les données dans les flux, s'il devient trop gros, il réinvente les fichiers d'échange pour une raison quelconque - dans le stockage isolé de l'utilisateur - et par coïncidence, la plupart des produits MS qui permettent Pour les compléments basés sur .NET, leurs domaines d'application ne sont pas configurés de telle sorte que vous puissiez accéder au stockage isolé. Les compléments VSTO sont connus pour souffrir de ce problème, par exemple.)

Answer 7

Voici une mise en œuvre complète:

/// <summary> 
/// Defines a MemoryStream that does not sit on the Large Object Heap, thus avoiding memory fragmentation. 
/// </summary> 
public sealed class ChunkedMemoryStream : Stream 
{ 
    /// <summary> 
    /// Defines the default chunk size. Currently defined as 0x10000. 
    /// </summary> 
    public const int DefaultChunkSize = 0x10000; // needs to be < 85000 

    private List<byte[]> _chunks = new List<byte[]>(); 
    private long _position; 
    private int _chunkSize; 
    private int _lastChunkPos; 
    private int _lastChunkPosIndex; 

    /// <summary> 
    /// Initializes a new instance of the <see cref="ChunkedMemoryStream"/> class. 
    /// </summary> 
    public ChunkedMemoryStream() 
     : this(DefaultChunkSize) 
    { 
    } 

    /// <summary> 
    /// Initializes a new instance of the <see cref="ChunkedMemoryStream"/> class. 
    /// </summary> 
    /// <param name="chunkSize">Size of the underlying chunks.</param> 
    public ChunkedMemoryStream(int chunkSize) 
     : this(null) 
    { 
    } 

    /// <summary> 
    /// Initializes a new instance of the <see cref="ChunkedMemoryStream"/> class based on the specified byte array. 
    /// </summary> 
    /// <param name="buffer">The array of unsigned bytes from which to create the current stream.</param> 
    public ChunkedMemoryStream(byte[] buffer) 
     : this(DefaultChunkSize, buffer) 
    { 
    } 

    /// <summary> 
    /// Initializes a new instance of the <see cref="ChunkedMemoryStream"/> class based on the specified byte array. 
    /// </summary> 
    /// <param name="chunkSize">Size of the underlying chunks.</param> 
    /// <param name="buffer">The array of unsigned bytes from which to create the current stream.</param> 
    public ChunkedMemoryStream(int chunkSize, byte[] buffer) 
    { 
     FreeOnDispose = true; 
     ChunkSize = chunkSize; 
     _chunks.Add(new byte[chunkSize]); 
     if (buffer != null) 
     { 
      Write(buffer, 0, buffer.Length); 
      Position = 0; 
     } 
    } 

    /// <summary> 
    /// Gets or sets a value indicating whether to free the underlying chunks on dispose. 
    /// </summary> 
    /// <value><c>true</c> if [free on dispose]; otherwise, <c>false</c>.</value> 
    public bool FreeOnDispose { get; set; } 

    /// <summary> 
    /// Releases the unmanaged resources used by the <see cref="T:System.IO.Stream"/> and optionally releases the managed resources. 
    /// </summary> 
    /// <param name="disposing">true to release both managed and unmanaged resources; false to release only unmanaged resources.</param> 
    protected override void Dispose(bool disposing) 
    { 
     if (FreeOnDispose) 
     { 
      if (_chunks != null) 
      { 
       _chunks = null; 
       _chunkSize = 0; 
       _position = 0; 
      } 
     } 
     base.Dispose(disposing); 
    } 

    /// <summary> 
    /// When overridden in a derived class, clears all buffers for this stream and causes any buffered data to be written to the underlying device. 
    /// This implementation does nothing. 
    /// </summary> 
    public override void Flush() 
    { 
     // do nothing 
    } 

    /// <summary> 
    /// When overridden in a derived class, reads a sequence of bytes from the current stream and advances the position within the stream by the number of bytes read. 
    /// </summary> 
    /// <param name="buffer">An array of bytes. When this method returns, the buffer contains the specified byte array with the values between <paramref name="offset"/> and (<paramref name="offset"/> + <paramref name="count"/> - 1) replaced by the bytes read from the current source.</param> 
    /// <param name="offset">The zero-based byte offset in <paramref name="buffer"/> at which to begin storing the data read from the current stream.</param> 
    /// <param name="count">The maximum number of bytes to be read from the current stream.</param> 
    /// <returns> 
    /// The total number of bytes read into the buffer. This can be less than the number of bytes requested if that many bytes are not currently available, or zero (0) if the end of the stream has been reached. 
    /// </returns> 
    /// <exception cref="T:System.ArgumentException"> 
    /// The sum of <paramref name="offset"/> and <paramref name="count"/> is larger than the buffer length. 
    /// </exception> 
    /// <exception cref="T:System.ArgumentNullException"> 
    ///  <paramref name="buffer"/> is null. 
    /// </exception> 
    /// <exception cref="T:System.ArgumentOutOfRangeException"> 
    ///  <paramref name="offset"/> or <paramref name="count"/> is negative. 
    /// </exception> 
    /// <exception cref="T:System.ObjectDisposedException"> 
    /// Methods were called after the stream was closed. 
    /// </exception> 
    public override int Read(byte[] buffer, int offset, int count) 
    { 
     if (buffer == null) 
      throw new ArgumentNullException("buffer"); 

     if (offset < 0) 
      throw new ArgumentOutOfRangeException("offset"); 

     if (count < 0) 
      throw new ArgumentOutOfRangeException("count"); 

     if ((buffer.Length - offset) < count) 
      throw new ArgumentException(null, "count"); 

     CheckDisposed(); 

     int chunkIndex = (int)(_position/ChunkSize); 
     if (chunkIndex == _chunks.Count) 
      return 0; 

     int chunkPos = (int)(_position % ChunkSize); 
     count = (int)Math.Min(count, Length - _position); 
     if (count == 0) 
      return 0; 

     int left = count; 
     int inOffset = offset; 
     int total = 0; 
     do 
     { 
      int toCopy = Math.Min(left, ChunkSize - chunkPos); 
      Buffer.BlockCopy(_chunks[chunkIndex], chunkPos, buffer, inOffset, toCopy); 
      inOffset += toCopy; 
      left -= toCopy; 
      total += toCopy; 
      if ((chunkPos + toCopy) == ChunkSize) 
      { 
       if (chunkIndex == (_chunks.Count - 1)) 
       { 
        // last chunk 
        break; 
       } 
       chunkPos = 0; 
       chunkIndex++; 
      } 
      else 
      { 
       chunkPos += toCopy; 
      } 
     } 
     while (left > 0); 
     _position += total; 
     return total; 
    } 

    /// <summary> 
    /// Reads a byte from the stream and advances the position within the stream by one byte, or returns -1 if at the end of the stream. 
    /// </summary> 
    /// <returns> 
    /// The unsigned byte cast to an Int32, or -1 if at the end of the stream. 
    /// </returns> 
    /// <exception cref="T:System.ObjectDisposedException"> 
    /// Methods were called after the stream was closed. 
    /// </exception> 
    public override int ReadByte() 
    { 
     CheckDisposed(); 
     if (_position >= Length) 
      return -1; 

     byte b = _chunks[(int)(_position/ChunkSize)][_position % ChunkSize]; 
     _position++; 
     return b; 
    } 

    /// <summary> 
    /// When overridden in a derived class, sets the position within the current stream. 
    /// </summary> 
    /// <param name="offset">A byte offset relative to the <paramref name="origin"/> parameter.</param> 
    /// <param name="origin">A value of type <see cref="T:System.IO.SeekOrigin"/> indicating the reference point used to obtain the new position.</param> 
    /// <returns> 
    /// The new position within the current stream. 
    /// </returns> 
    /// <exception cref="T:System.ObjectDisposedException"> 
    /// Methods were called after the stream was closed. 
    /// </exception> 
    public override long Seek(long offset, SeekOrigin origin) 
    { 
     CheckDisposed(); 
     switch (origin) 
     { 
      case SeekOrigin.Begin: 
       Position = offset; 
       break; 

      case SeekOrigin.Current: 
       Position += offset; 
       break; 

      case SeekOrigin.End: 
       Position = Length + offset; 
       break; 
     } 
     return Position; 
    } 

    private void CheckDisposed() 
    { 
     if (_chunks == null) 
      throw new ObjectDisposedException(null, "Cannot access a disposed stream"); 
    } 

    /// <summary> 
    /// When overridden in a derived class, sets the length of the current stream. 
    /// </summary> 
    /// <param name="value">The desired length of the current stream in bytes.</param> 
    /// <exception cref="T:System.ObjectDisposedException"> 
    /// Methods were called after the stream was closed. 
    /// </exception> 
    public override void SetLength(long value) 
    { 
     CheckDisposed(); 
     if (value < 0) 
      throw new ArgumentOutOfRangeException("value"); 

     if (value > Length) 
      throw new ArgumentOutOfRangeException("value"); 

     long needed = value/ChunkSize; 
     if ((value % ChunkSize) != 0) 
     { 
      needed++; 
     } 

     if (needed > int.MaxValue) 
      throw new ArgumentOutOfRangeException("value"); 

     if (needed < _chunks.Count) 
     { 
      int remove = (int)(_chunks.Count - needed); 
      for (int i = 0; i < remove; i++) 
      { 
       _chunks.RemoveAt(_chunks.Count - 1); 
      } 
     } 
     _lastChunkPos = (int)(value % ChunkSize); 
    } 

    /// <summary> 
    /// Converts the current stream to a byte array. 
    /// </summary> 
    /// <returns>An array of bytes</returns> 
    public byte[] ToArray() 
    { 
     CheckDisposed(); 
     byte[] bytes = new byte[Length]; 
     int offset = 0; 
     for (int i = 0; i < _chunks.Count; i++) 
     { 
      int count = (i == (_chunks.Count - 1)) ? _lastChunkPos : _chunks[i].Length; 
      if (count > 0) 
      { 
       Buffer.BlockCopy(_chunks[i], 0, bytes, offset, count); 
       offset += count; 
      } 
     } 
     return bytes; 
    } 

    /// <summary> 
    /// When overridden in a derived class, writes a sequence of bytes to the current stream and advances the current position within this stream by the number of bytes written. 
    /// </summary> 
    /// <param name="buffer">An array of bytes. This method copies <paramref name="count"/> bytes from <paramref name="buffer"/> to the current stream.</param> 
    /// <param name="offset">The zero-based byte offset in <paramref name="buffer"/> at which to begin copying bytes to the current stream.</param> 
    /// <param name="count">The number of bytes to be written to the current stream.</param> 
    /// <exception cref="T:System.ArgumentException"> 
    /// The sum of <paramref name="offset"/> and <paramref name="count"/> is greater than the buffer length. 
    /// </exception> 
    /// <exception cref="T:System.ArgumentNullException"> 
    ///  <paramref name="buffer"/> is null. 
    /// </exception> 
    /// <exception cref="T:System.ArgumentOutOfRangeException"> 
    ///  <paramref name="offset"/> or <paramref name="count"/> is negative. 
    /// </exception> 
    /// <exception cref="T:System.ObjectDisposedException"> 
    /// Methods were called after the stream was closed. 
    /// </exception> 
    public override void Write(byte[] buffer, int offset, int count) 
    { 
     if (buffer == null) 
      throw new ArgumentNullException("buffer"); 

     if (offset < 0) 
      throw new ArgumentOutOfRangeException("offset"); 

     if (count < 0) 
      throw new ArgumentOutOfRangeException("count"); 

     if ((buffer.Length - offset) < count) 
      throw new ArgumentException(null, "count"); 

     CheckDisposed(); 

     int chunkPos = (int)(_position % ChunkSize); 
     int chunkIndex = (int)(_position/ChunkSize); 
     if (chunkIndex == _chunks.Count) 
     { 
      _chunks.Add(new byte[ChunkSize]); 
     } 

     int left = count; 
     int inOffset = offset; 
     do 
     { 
      int copied = Math.Min(left, ChunkSize - chunkPos); 
      Buffer.BlockCopy(buffer, inOffset, _chunks[chunkIndex], chunkPos, copied); 
      inOffset += copied; 
      left -= copied; 
      if ((chunkPos + copied) == ChunkSize) 
      { 
       chunkIndex++; 
       chunkPos = 0; 
       if (chunkIndex == _chunks.Count) 
       { 
        _chunks.Add(new byte[ChunkSize]); 
       } 
      } 
      else 
      { 
       chunkPos += copied; 
      } 
     } 
     while (left > 0); 
     _position += count; 
     if (chunkIndex == (_chunks.Count - 1)) 
     { 
      if ((chunkIndex > _lastChunkPosIndex) || 
       ((chunkIndex == _lastChunkPosIndex) && (chunkPos > _lastChunkPos))) 
      { 
       _lastChunkPos = chunkPos; 
       _lastChunkPosIndex = chunkIndex; 
      } 
     } 
    } 

    /// <summary> 
    /// Writes a byte to the current position in the stream and advances the position within the stream by one byte. 
    /// </summary> 
    /// <param name="value">The byte to write to the stream.</param> 
    /// <exception cref="T:System.ObjectDisposedException"> 
    /// Methods were called after the stream was closed. 
    /// </exception> 
    public override void WriteByte(byte value) 
    { 
     CheckDisposed(); 
     int chunkIndex = (int)(_position/ChunkSize); 
     int chunkPos = (int)(_position % ChunkSize); 

     if (chunkPos > (ChunkSize - 1)) //changed from (chunkPos >= (ChunkSize - 1)) 
     { 
      chunkIndex++; 
      chunkPos = 0; 
      if (chunkIndex == _chunks.Count) 
      { 
       _chunks.Add(new byte[ChunkSize]); 
      } 
     } 
     _chunks[chunkIndex][chunkPos++] = value; 
     _position++; 
     if (chunkIndex == (_chunks.Count - 1)) 
     { 
      if ((chunkIndex > _lastChunkPosIndex) || 
       ((chunkIndex == _lastChunkPosIndex) && (chunkPos > _lastChunkPos))) 
      { 
       _lastChunkPos = chunkPos; 
       _lastChunkPosIndex = chunkIndex; 
      } 
     } 
    } 

    /// <summary> 
    /// Writes to the specified stream. 
    /// </summary> 
    /// <param name="stream">The stream.</param> 
    public void WriteTo(Stream stream) 
    { 
     if (stream == null) 
      throw new ArgumentNullException("stream"); 

     CheckDisposed(); 
     for (int i = 0; i < _chunks.Count; i++) 
     { 
      int count = (i == (_chunks.Count - 1)) ? _lastChunkPos : _chunks[i].Length; 
      stream.Write(_chunks[i], 0, count); 
     } 
    } 

    /// <summary> 
    /// When overridden in a derived class, gets a value indicating whether the current stream supports reading. 
    /// </summary> 
    /// <value></value> 
    /// <returns>true if the stream supports reading; otherwise, false. 
    /// </returns> 
    public override bool CanRead 
    { 
     get 
     { 
      return true; 
     } 
    } 

    /// <summary> 
    /// When overridden in a derived class, gets a value indicating whether the current stream supports seeking. 
    /// </summary> 
    /// <value></value> 
    /// <returns>true if the stream supports seeking; otherwise, false. 
    /// </returns> 
    public override bool CanSeek 
    { 
     get 
     { 
      return true; 
     } 
    } 

    /// <summary> 
    /// When overridden in a derived class, gets a value indicating whether the current stream supports writing. 
    /// </summary> 
    /// <value></value> 
    /// <returns>true if the stream supports writing; otherwise, false. 
    /// </returns> 
    public override bool CanWrite 
    { 
     get 
     { 
      return true; 
     } 
    } 

    /// <summary> 
    /// When overridden in a derived class, gets the length in bytes of the stream. 
    /// </summary> 
    /// <value></value> 
    /// <returns> 
    /// A long value representing the length of the stream in bytes. 
    /// </returns> 
    /// <exception cref="T:System.ObjectDisposedException"> 
    /// Methods were called after the stream was closed. 
    /// </exception> 
    public override long Length 
    { 
     get 
     { 
      CheckDisposed(); 
      if (_chunks.Count == 0) 
       return 0; 

      return (_chunks.Count - 1) * ChunkSize + _lastChunkPos; 
     } 
    } 

    /// <summary> 
    /// Gets or sets the size of the underlying chunks. Cannot be greater than or equal to 85000. 
    /// </summary> 
    /// <value>The chunks size.</value> 
    public int ChunkSize 
    { 
     get 
     { 
      return _chunkSize; 
     } 
     set 
     { 
      if ((value <= 0) || (value >= 85000)) 
       throw new ArgumentOutOfRangeException("value"); 

      _chunkSize = value; 
     } 
    } 

    /// <summary> 
    /// When overridden in a derived class, gets or sets the position within the current stream. 
    /// </summary> 
    /// <value></value> 
    /// <returns> 
    /// The current position within the stream. 
    /// </returns> 
    /// <exception cref="T:System.ObjectDisposedException"> 
    /// Methods were called after the stream was closed. 
    /// </exception> 
    public override long Position 
    { 
     get 
     { 
      CheckDisposed(); 
      return _position; 
     } 
     set 
     { 
      CheckDisposed(); 
      if (value < 0) 
       throw new ArgumentOutOfRangeException("value"); 

      if (value > Length) 
       throw new ArgumentOutOfRangeException("value"); 

      _position = value; 
     } 
    } 
} 

Answer 8

L'équipe Bing a publié RecyclableMemoryStream et écrit à ce sujet here. Les avantages mentionnés sont:

1. Éliminez les grandes allocations de tas d'objets à l'aide de tampons mis en commun
2. SUBIR beaucoup moins de gen 2 GCS, et passer beaucoup moins de temps mis en pause en raison de GC
3. fuites de mémoire Évitez en ayant un bornées taille de la piscine
4. Éviter la fragmentation de la mémoire
5. est extrêmement bien debuggability
6. fournissent des mesures pour le suivi des performances

Answer 9

Une autre implémentation de flux segmenté pourrait être considérée comme un remplacement MemoryStream en stock. En outre, il permet d'affecter un seul grand tableau d'octets sur LOH qui sera utilisé comme « morceau » piscine, partagée entre toutes les instances de ChunkedStream ...

https://github.com/ImmortalGAD/ChunkedStream