Setup: I have a requirement to handle large file uploading from a web client to sql database. I can't write to the file system so I can't use the out of the box MultipartFormDataStreamProvider and I can't load the files into memory because they are large (over 1gb). The requirement I have is to take the parts, encrypt them, and store the part in a row in a sql db. Later another process will take those parts, decrypt them, recombine them and transmit the file to another system that will process it accordingly.
Issue: I have gotten the data uploaded to SQL but what is being stored is the entire request (including headers) so when I pull the data back and recombine them, they are resulting in corrupt files. I can get it to work with text files
if I parse the buffer directly and remove the headers themselves. But this methodology fails when non-text files are uploaded (pdf, zip, img, etc). I've been spinning my wheels on this for a few days now and I'm just stumped as to what I'm doing wrong.
Any help would be greatly appreciated!
Current Methodology: My methodology is to override the WebBuffer policy so that .net doesnt buffer the request (thus removing any out of memory issues), overriding the GetStream method of a custom class that inherits MultipartFormDataStreamProvider and
returning a custom stream (inherited Stream and overrode the Write method so that it encrypts the buffer and writes the encrypted buffer to sql).
The CustomMultipartFormDataStreamProvider is very generalized and just returns a type of CustomSqlStream when headers.ContentDisposition.Filename isnt empty and returns a new MemoryStream otherwise. The code is:
public override Stream GetStream(HttpContent parent, HttpContentHeaders headers) { // If we have a file name then write contents out to custom stream. Otherwise just write to MemoryStream if (headers.ContentType != null && headers.ContentDisposition != null && !string.IsNullOrEmpty(headers.ContentDisposition.FileName)) { // For form data, Content-Disposition header is a requirement ContentDispositionHeaderValue contentDisposition = headers.ContentDisposition; var identifier = Guid.NewGuid().ToString(); var fileName = contentDisposition.FileName;// GetLocalFileName(headers); var boundaryObj = parent.Headers.ContentType.Parameters.SingleOrDefault(a => a.Name == "boundary"); var boundary = (boundaryObj != null) ? boundaryObj.Value : ""; if (fileName.Contains("\\")) { fileName = fileName.Substring(fileName.LastIndexOf("\\") + 1).Replace("\"", ""); } // We won't post process files as form data _isFormData.Add(false); var stream = new CustomSqlStream(); stream.Filename = fileName; stream.Identifier = identifier; stream.ContentType = headers.ContentType.MediaType; stream.Boundary = (!string.IsNullOrEmpty(boundary)) ? boundary : ""; return stream; throw new InvalidOperationException("Did not find required 'Content-Disposition' header field in MIME multipart body part.."); } // We will post process this as form data _isFormData.Add(true); // If no filename parameter was found in the Content-Disposition header then return a memory stream. return new MemoryStream(); }
The API method called by the client is:
public Task<HttpResponseMessage> PostFormData() { var provider = new CustomMultipartFormDataStreamProvider(); // Read the form data and return an async task. var task = Request.Content.ReadAsMultipartAsync(provider).ContinueWith<HttpResponseMessage>(t => { if (t.IsFaulted || t.IsCanceled) { Request.CreateErrorResponse(HttpStatusCode.InternalServerError, t.Exception); } return Request.CreateResponse(HttpStatusCode.OK); }); return task; }
My current CustomSqlStream.Write method is (this.Boundary is assigned to the derived Stream class when it's created. It's pulled from the headers of the multipart request):
public override void Write(byte[] buffer, int offset, int count) { string formData = Encoding.UTF8.GetString(buffer); bool trimmed = false; //check for boundary if (formData.Contains(this.Boundary)) { var endPattern = String.Format("{0}{1}{2}", "\r\n--", this.Boundary, "--\r\n"); if (formData.Contains(endPattern)) { //this is a end of boundary occurrence formData = formData.Substring(0, formData.IndexOf(endPattern)); trimmed = true; } if (formData.Contains(this.Boundary)) { //this is a header data occurrence var boundaryOffset = formData.IndexOf(this.ContentType) + this.ContentType.Length + "\r\n\r\n".Length; formData = formData.Substring(boundaryOffset); trimmed = true; } } if (trimmed) { byte[] body = new byte[formData.Length]; buffer = Encoding.UTF8.GetBytes(formData); for (int j = 0, k = 0; j < formData.Length; j++, k++) { body[k] = buffer[j]; } WriteData(body); } else { WriteData(buffer); } _dataAddedEvent.Set(); }