AWS: Simple Storage Service (S3) Helper Class - C#

As per Amazon, Amazon Simple Storage Service (S3) is storage for the Internet. It is designed to make web-scale computing easier for developers. It has a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly salable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers.

Following is the helper class for working with the objects stored in Amazon S3 buckets using C#.NET. AWSSDK.S3 (https://www.nuget.org/packages/AWSSDK.S3/) is the NuGet package required to work with AWS S3.

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301

public class AmazonS3Helper {  AmazonS3Client client;    private static readonly ILog _logger = LogManager.GetLogger(typeof(AmazonS3Helper));  private readonly string accessKeyId, secretKey, serviceUrl;  public AmazonS3Helper(string accessKeyId, string secretKey, string serviceUrl)  {   this.accessKeyId = accessKeyId;   this.secretKey = secretKey;   this.serviceUrl = serviceUrl;   client = GetClient();  }    /// <summary>  /// Initializes and returns the AmazonS3 object  /// </summary>  /// <returns></returns>  private AmazonS3Client GetClient()  {   if (client == null)   {    try    {     // S3 config object     AmazonS3Config clientConfig = new AmazonS3Config     {      // Set the endpoint URL      ServiceURL = serviceUrl     };     client = new AmazonS3Client(accessKeyId, secretKey, clientConfig);    }    catch (AmazonS3Exception ex)    { _logger.Error($"Error (AmazonS3Exception) creating S3 client", ex); }    catch (AmazonServiceException ex)    { _logger.Error($"Error (AmazonServiceException) creating S3 client", ex); }    catch (Exception ex)    { _logger.Error($"Error creating AWS S3 client", ex); }   }   return client;  }    /// <summary>  /// Uploads the file to the S3 bucket.  /// </summary>  /// <param name="bucketNameWithPath">S3 bucket name along with the subfolders. Ex. If you are using a 'dev' folder under bucket 'myBucket', this value should be myBucket/dev</param>  /// <param name="fileNameInS3">File name used to store the content in the bucket</param>  /// <param name="fileContent">String content which needs to be stored in the file</param>  /// <returns></returns>  public async Task Upload(string bucketNameWithPath, string fileNameInS3, string fileContent)  {   _logger.Info("Entering AmazonS3Helper.Upload");   try   {    byte[] byteArray = Encoding.ASCII.GetBytes(fileContent);    MemoryStream stream = new MemoryStream(byteArray);      var putRequest = new PutObjectRequest    {     BucketName = bucketNameWithPath,     Key = fileNameInS3,     InputStream = stream    };    await client.PutObjectAsync(putRequest);   }   catch (Exception ex)   {    _logger.Error("Error in uploading file to s3 bucket", ex);   }     _logger.Info("Leaving AmazonS3Helper.Upload");  }      /// <summary>  /// Downloads the file from S3 bucket  /// </summary>  /// <param name="bucketNameWithPath">S3 bucket name along with the subfolders. Ex. If you are using a 'dev' folder under bucket 'myBucket', this value should be myBucket/dev</param>  /// <param name="fileNameInS3">File which needs to be get from the bucket</param>  /// <param name="downloadFolder">Local folder path to store the file downloaded from S3</param>  /// <returns></returns>  public async Task Download(string bucketNameWithPath, string fileNameInS3, string downloadFolder)  {   _logger.Info("Entering AmazonS3Helper.Download");   try   {    var getRequest = new GetObjectRequest    {     BucketName = bucketNameWithPath,     Key = fileNameInS3    };    using (GetObjectResponse response = await client.GetObjectAsync(getRequest))    using (StreamReader reader = new StreamReader(response.ResponseStream))    {     var fileBody = reader.ReadToEnd(); // Now you process the response body.     if (!Directory.Exists(downloadFolder))      Directory.CreateDirectory(downloadFolder);     // Append filename     downloadFolder = downloadFolder + "/" + fileNameInS3;       File.WriteAllText(downloadFolder, fileBody);    }   }   catch (Exception ex)   {    _logger.Error("Error in downloading file to s3 bucket", ex);   }     _logger.Info("Leaving AmazonS3Helper.Download");  }    /// <summary>  /// Copies a file from one bucket to another bucket  /// </summary>  /// <param name="sourceBucketWithPath">Source bucket name along with the subfolders. Ex. If you are using a 'dev' folder under bucket 'myBucket', this value should be myBucket/dev</param>  /// <param name="sourceFileName">File name in source bucket</param>  /// <param name="destinationBucketWithPath">Destination bucket name along with the subfolders. Ex. If you are using a 'dev' folder under bucket 'myBucket', this value should be myBucket/dev</param>  /// <param name="destinationFileName">File name in destination bucket</param>  /// <returns></returns>  public async Task Copy(string sourceBucketWithPath, string sourceFileName, string destinationBucketWithPath, string destinationFileName)  {   _logger.Info("Entering AmazonS3Helper.Copy");   try   {    var copyRequest = new CopyObjectRequest    {     SourceBucket = sourceBucketWithPath,     SourceKey = sourceFileName,     DestinationBucket = destinationBucketWithPath,     DestinationKey = destinationFileName    };    await client.CopyObjectAsync(copyRequest);   }   catch (Exception ex)   {    _logger.Error("Error in copying file in S3", ex);   }     _logger.Info("Leaving AmazonS3Helper.Copy");  }    /// <summary>  /// Lists all the files in the given bucket  /// </summary>  /// <param name="bucketName">S3 bucket name </param>  /// <param name="path">Folder path in the bucket</param>  /// <returns></returns>  public async Task<List<S3Object>> ListFiles(string bucketName, string path = "")  {   List<S3Object> files = new List<S3Object>();   _logger.Info("Entering AmazonS3Helper.ListFiles");   try   {    var listRequest = new ListObjectsV2Request    {     BucketName = bucketName,     MaxKeys = 10    };      if (!string.IsNullOrEmpty(path))    {     if (path[path.Length - 1] != '/')      path += "/";     listRequest.Prefix = path;    }      ListObjectsV2Response response;    do    {     response = await client.ListObjectsV2Async(listRequest);     files.AddRange(response.S3Objects);     listRequest.ContinuationToken = response.NextContinuationToken;    } while (response.IsTruncated);       }   catch (Exception ex)   {    _logger.Error("Error while listing files in S3", ex);   }   _logger.Info("Leaving AmazonS3Helper.ListFiles");   return files;  }    /// <summary>  /// Deletes the file from S3 bucket  /// </summary>  /// <param name="bucketNameWithPath">S3 bucket name along with the subfolders. Ex. If you are using a 'dev' folder under bucket 'myBucket', this value should be myBucket/dev</param>  /// <param name="fileNameInS3">File which needs to be delete from the bucket</param>  /// <returns></returns>  public async Task Delete(string bucketNameWithPath, string fileNameInS3)  {   _logger.Info("Entering AmazonS3Helper.Delete");   try   {    var deleteObjectRequest = new DeleteObjectRequest    {     BucketName = bucketNameWithPath,     Key = fileNameInS3    };      await client.DeleteObjectAsync(deleteObjectRequest);   }   catch (Exception ex)   {    _logger.Error("Error in deleting file from s3 bucket", ex);   }     _logger.Info("Leaving AmazonS3Helper.Delete");  }    /// <summary>  /// Deletes multiple files from S3 bucket  /// </summary>  /// <param name="bucketName">S3 bucket name</param>  /// <param name="path">Folder path in the bucket</param>  /// <returns></returns>  public async Task<int> DeleteFolder(string bucketName, string path)  {   var fileDeleted = 0;   _logger.Info("Entering AmazonS3Helper.MultipleDelete");   try   {    var files = await ListFiles(bucketName, path);      // Converting filesnames as KeyVersions List    List<KeyVersion> keyVersions = files.Select(x => new KeyVersion() { Key = x.Key }).ToList();      var deleteObjectsRequest = new DeleteObjectsRequest    {     BucketName = bucketName,     Objects = keyVersions    };    DeleteObjectsResponse response = await client.DeleteObjectsAsync(deleteObjectsRequest);    fileDeleted = response.DeletedObjects.Count;   }   catch (DeleteObjectsException e)   {    _logger.Error("Error in deleting files from s3 bucket");    // Catch the successfully deleted files count    fileDeleted = e.Response.DeletedObjects.Count;    foreach (DeleteError deleteError in e.Response.DeleteErrors)    {     _logger.Error($"Object Key: {deleteError.Key}\t{deleteError.Code}\t{deleteError.Message}");    }   }   catch (Exception ex)   {    _logger.Error("Error in deleting files from s3 bucket", ex);   }   _logger.Info("Leaving AmazonS3Helper.MultipleDelete");   return fileDeleted;  }    /// <summary>  /// Deletes multiple files from S3 bucket  /// </summary>  /// <param name="bucketName">S3 bucket name</param>  /// <param name="path">Folder path in the bucket</param>  /// <param name="fileNamesInS3">Files which needs to be delete from the bucket</param>  /// <returns></returns>  public async Task<int> Delete(string bucketName,  string[] fileNamesInS3, string path="")  {   var fileDeleted = 0;   _logger.Info("Entering AmazonS3Helper.MultipleDelete");   try   {    if (!string.IsNullOrEmpty(path))    {     if (path[path.Length - 1] != '/')      path += "/";    }    // Converting filesnames as KeyVersions List    List<KeyVersion> keyVersions = fileNamesInS3.Select(x => new KeyVersion() { Key = path + x }).ToList();      var deleteObjectsRequest = new DeleteObjectsRequest    {     BucketName = bucketName,     Objects = keyVersions    };    DeleteObjectsResponse response = await client.DeleteObjectsAsync(deleteObjectsRequest);    fileDeleted = response.DeletedObjects.Count;   }   catch (DeleteObjectsException e)   {    _logger.Error("Error in deleting files from s3 bucket");    // Catch the successfully deleted files count    fileDeleted = e.Response.DeletedObjects.Count;    foreach (DeleteError deleteError in e.Response.DeleteErrors)    {     _logger.Error($"Object Key: {deleteError.Key}\t{deleteError.Code}\t{deleteError.Message}");    }   }   catch (Exception ex)   {    _logger.Error("Error in deleting files from s3 bucket", ex);   }   _logger.Info("Leaving AmazonS3Helper.MultipleDelete");   return fileDeleted;  } }

Usage:

Create the helper class object

?

1	AmazonS3Helper amazonS3Helper = new AmazonS3Helper(<accesskey>, <secret key>, <AWS S3 endpoint url>);

Upload a file
Here, I am uploading S3.txt file to the dev folder in my S3 bucket named gopiBucket with name "S3Sample_1.txt"

?

1 2	string fileContent = File.ReadAllText("S3.txt"); await amazonS3Helper.Upload("gopiBucket/dev", "S3Sample_1.txt", fileContent);

Copy File:
Copy file method copies the file from one bucket to another bucket or one folder to another folder in a bucket. Here, I am copying my file from dev folder to prod folder in same bucket

?

1	await amazonS3Helper.Copy("gopiBucket/dev", "S3Sample_1.txt", "gopiBucket/prod", "S3Sample_1_prod.txt");

List files:
Here I am listing all the files in prod folder.

?

1 2 3 4 5

var files = await amazonS3Helper.ListFiles("gopiBucket", "prod"); foreach (var file in files) {  Console.WriteLine($"{file.Key}\t{file.Size}"); }

Download a file:
Donwloading the file from dev folder to the "Downloads" folder in my solution. If the "Downloads" folder not exits, it will create new one.

?

1	await amazonS3Helper.Download("gopiBucket/dev", "S3Sample_1.txt", "Downloads");

Delete a file:

?

1	await amazonS3Helper.Delete("gopiBucket/prod", "S3Sample_1_prod.txt");

Delete multiple files:

?

1 2	string[] filesToDelete = new string[] { "S3Sample_1_prod.txt", "S3Sample_2_prod.txt", "S3Sample_3_prod.txt" }; await amazonS3Helper.Delete("gopiBucket", filesToDelete, "prod");

Delete a completed folder:

?

1	await amazonS3Helper.DeleteFolder("gopiBucket", "prod/S1")

Note: The above mentioned Upload and Download operation methods uploads / downloads the files using a single GET / PUT operation. So you can upload / download objects up to 5 GB in size. If you want to store / retrieve more than 5GB, you need to use Multipart Upload API. I will give the Multipart Upload API details in my next post.

Happy Coding 😊 !!

Gopikrishna