PowerShell command to get the metadata

Here is the command to get the metadata of all of your blobs in your storage account.

Set your execution policy to Process if you get any policy errors.

Set-ExecutionPolicy -ExecutionPolicy ByPass -Scope Process

If you get an authorization error as follows, make sure to add a role assignment of Storage Blob Data Contributor. You can do this via Access Control (IAM) of your storage account. Read more about this role here.

PowerShell AuthorizationPermissionMismatch Error

Get-AzStorageContainer : This request is not authorized to perform this operation using this permission.
RequestId:405a7e1a-f01e-004f-3acd-6a1168000000
Time:2022-05-18T15:38:30.5034936Z
Status: 403 (This request is not authorized to perform this operation using this permission.)
ErrorCode: AuthorizationPermissionMismatch
Content:
<?xml version=”1.0″ encoding=”utf-8″?><Error><Code>AuthorizationPermissionMismatch</Code><Message>This request is not authorized to perform this operation using
this permission.

Once you add the role, you can try running your PowerShell command after a few minutes. Please make sure to provide the values to the variables $containerName, $subscriptionid, $storageAccName. Happy coding with PowerShell!

About the Author

I am yet another developer who is passionate about writing and video creation. I have written close to 500 blogs on my blog. If you like this content, consider following me here,

• GitHub
• medium
• Twitter

Your turn. What do you think?

Thanks a lot for reading. Did I miss anything that you may think is needed in this article? Could you find this post useful? Kindly do not forget to share your feedback.

Kindest Regards

Sibeesh Venu

Introduction

In our previous article we have already seen that how we can use Quartz.Net scheduler in our Asp.Net core console application and then run it as a windows service. The problem with that approach is that, the scheduler information will be lost once the system restarts or something bad happens. To rectify that problem we can actually save all the scheduler information in a database, luckily the Quartz.Net scheduler supports this and we can easily implement it. I hope the introduction is clear and let’s jump on to the implementation.

Source Code

The source code can be found in this GitHub repository. Please feel free to star, fork or do whatever you wish. You can also consider follow me on GitHub. LOL.

Setting Up Database

Before we start the coding, let us set up our database. You can choose any database as you wish. I am choosing SQL Server Express.

Install SQL Server Express

Enable sa Login

We should enable the sa login which we will use in our Quartz configuration. If you try to configure the Quartz with windows authentication, you will get an error as below.

2019-08-29 11:34:02.3570|INFO|Quartz.Core.QuartzScheduler|JobFactory set to: Backup.Service.Helpers.CustomJobFactory
2019-08-29 11:34:12.6801|INFO|Quartz.Impl.AdoJobStore.JobStoreTX|ConnectionAndTransactionHolder passed to RollbackConnection was null, ignoring
2019-08-29 11:34:12.6892|ERROR|Quartz.Impl.AdoJobStore.JobStoreTX|Failure occurred during job recovery: Failed to obtain DB connection from data source ‘default’: System.Data.SqlClient.SqlException (0x80131904): Cannot open database “BackupClientSchedules” requested by the login. The login failed.
Login failed for user ‘WORKGROUP\DESKTOP-3EF5B65$’.
at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling)
at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection) at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions) at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource1 retry)
at System.Data.SqlClient.SqlConnection.Open()
at Quartz.Impl.AdoJobStore.JobStoreSupport.GetConnection() in C:\projects\quartznet\src\Quartz\Impl\AdoJobStore\JobStoreSupport.cs:line 323
ClientConnectionId:2efd5ab8-71aa-4598-b5ca-63c5aaa6afbe
Error Number:4060,State:1,Class:11

Yes, that is such a log error, and we don’t want any errors right. So let us configure sa login first. I have already explained how you can do that , in one of my StackOverflow answer, you can see that here.

Create Database and Tables Needed

Now we need to create a database and some tables where we can save the scheduler and trigger information. You can always see the table schema here in this official Quartz GitHub repository. You can also find the same file in our source code.

Before you run that query in your SQL Server Management Studio, make sure to create a database first and the use the same.

CREATE database BackupClientSchedules
USE BackupClientSchedules

Now id you see inside the database BackupClientSchedules, you should see some tables with no data inside.

Configure Quartz.Net to Use Database Instead of RAM

As we have already set up our database, now we can start configuring the service. If you are new Quartz.Net scheduler, I strongly recommend you to read my previous article.

Edit App.config

Let’s change the App.config file as preceding.

Note that we have set quartz.dataSource.default.provider as Sqlserver and also given the quartz.dataSource.default.connectionString. You should change these values with your provider and connection string.

Install the Nuget Package

We should install the Nuget package Quartz.Serialization.Json in our application.

Setup the Scheduler

In our previous article, we have see how we can setup the scheduler and we were using a function called GetScheduler for the same. Now we can edit the code of that function as preceding.

Please note the comments in the above code block.

Install the Service

Once we finished the above steps, we can run dotnet publish command with release configuration and install the service as preceding.

Again, read my previous article, if you need a detailed explanations about this step.

Output

Once the service started successfully, you should be able to see two log files backupclientlogfile_backupservice.txt and backupclientlogfile_helperservice.txt in our win7-x64 folder. You can see all the logs in this files.

You should also be able to see some scheduling information in our tables in our database. Some of the table’s screenshots are given below.

Conclusion

Wow!. Now we have learned,

• about Windows Service and Asp.Net Console Application
• about How to create a Windows Service using Asp.Net Core
• about How to use Quartz scheduler with database

Your turn. What do you think?

Thanks a lot for reading. Did I miss anything that you may think which is needed in this article? Could you find this post as useful? Kindly do not forget to share me your feedback.

Kindest Regards
Sibeesh Venu

Introduction

Windows services are a good way for reducing some manual jobs that we have to do in our system. In this article, we are going to write a windows service using Asp.Net core, the jobs of this windows service is given below.

1. Zip the folder and save the file to a particular directory
2. Upload the zipped folder to the Azure blob storage

The above mentioned tasks will be running daily, weekly, monthly. We are using Quartz scheduler with Dependency Injection to do this amazing tasks. We are using Nlog for logging the details of our tasks. If you think this is something that you are interested at, then you are at the right place, let us develop our PerfectScheduler.

Source Code

The source code of this project is been uploaded to the GitHub, please feel free to fork, star, create pull requests etc. The repository can be found here.

Creating Perfect Scheduler

As I mentioned earlier, we are creating a windows service with Asp.Net core, technically there is no straight way to do this, as the Windows Service with Asp.Net core template is not available in Visual Studio.

so we will be creating an Asp.Net console application and then install executable file generated as a windows service.

Creating an Asp.Net Console Application

Open Visual Studio and search for the project template Console App(.Net Core) and name the solution as per your convenience, I am naming this project as Perfect Scheduler, as I am thinking that we, the community people can make this windows service perfect by creating many pull requests.

Once you had created the application, let us install all of our dependencies at once, so that we don’t need to worry about them later.

Write the Service

As we have installed all of our dependencies, let us create our service now. Add a new class with the name BackupService and inherit the same from IHostedService, which is part of Microsoft.Extensions.Hosting namespace.

The IHostedService has two methods in it as follows, so we should implement them in our service class.

StartAsync

The StartAsync method can be implemented as preceding.

As you can see that in the first line of the above code, we are getting the scheduler of Quartz. Let us create a method and return a scheduler then.

The next step is to build a service provider so that we can inject our dependencies using Constructor Dependency Injection. By default the Quartz is not doing this, so we have to build the configuration our own.

As you can see that we have configured the services for DailyJob, WeeklyJob, MonthlyJob and HelperService. No worries, we will be creating these Classes and Interfaces very soon.

Once we get the Service Provider, we can pass this to our Custom Job Factory, which we will be implementing soon. Now, we can start the scheduler and schedule our jobs, please make sure that you are using different Identity names for both Triggers and Jobs. The samples are given below.

StopAsync

The StopAsync method can be implemented as follows

Creating the JobBuilders and TriggerBuilders

Now we can create the Interfaces and Classes for our Jobs, which are Daily, Weekly, Monthly. To do so, let us create a new folder called Helpers and then create a folder called Interfaces inside. Below are the Interfaces you need to create then.

IDailyJob

IWeeklyJob

IMonthlyJob

Let us also create an Interface for our HelperService.

IHelperService

The parameter schedule is for determining which job it is, the value of this parameter can be daily, weekly, monthly.

Create a Custom Job Factory

To implement the Dependency Injection, we need to create our own custom job factory. Create a class inside the Helper folder as preceding.

Implement the Job Builder Interfaces

Now it is time to implement our Daily, Weekly, Monthly job builders.

DailyJob

WeeklyJob

MonthlyJob

As you can see that we have configured separate classes for our each jobs with the dependency IHelperService injected via constructor. In the future we should be able to write custom logic for each jobs here as it is in separate classes.

Implement HelperService

Before we start implementing this service let us configure the NLog now as we will be writing logs from this class.

Configure NLog

To configure, NLog, create a new configuration file NLog.config and edit the content as preceding.

Now, create a method SetUpNLog() and add the codes as below.

Please make sure that you have already added a property ILogger _logger.

We can implement the HelperService as preceding with all the necessary private and public methods.

The method PerformService(string schedule) will be called for every schedule and it will make sure that the below tasks are performed.

1. Zip the folder and save the file to a particular directory
2. Upload the zipped file to the Azure blob storage

Here, the values of the the variable and the blob storage container names are same, that is, daily, weekly, monthly. If you are not sure how you can upload a blob to Azure storage account, I strongly recommend you to read my article here.

Remember to set the values for the FolderToZipLocation(To which location the file should be saved), FolderFromZipLocation (From where the files should be taken), StorageConnectionString in the App.config file.

Setting Up The Program

As you all know that Program class it the start of our console application, not it is time to call our BackupService from the Program class. Let us edit the code of the Program class as below.

If we are running the application locally/debug, we are calling the extension method RunConsoleAsync() or else we call our own custom extension method RunTheServiceAsync(). The line services.AddHostedService() is very important as this is where we register our IHostedService, which is BackupService. Below is the code for our extension method.

The ServiceLifetime class is where we override the methods from ServiceBase class. You can find the implementation as follows.

Creating the Windows Service

As we have already created the Asp.Net Core console application, now it is time to create a windows service from it. Make sure that you had set the RuntimeIdentifier to win7-x64 and SelfContained to true in the properties of your project, this will make sure that all of your dependencies are being added to your executable file, so that you don’t need to worry about handling your dependencies manually. You can read more about it here. To do so, right click on the project and click on Edit Project File. At the end, your csproj file should be similar as preceding.

Create the Release Configuration

You should also run the dotnet publish command with the release configuration before you try to install the service, because you need this executable file to install the service.

Open the command prompt with administrator privilege and go to the project root folder using the cd command. Run the command “dotnet publish –configuration=release” .

This will generate everything you wanted. If you run the command correctly, you should see an output as preceding.

Now go to the bin folder and then release folder, you should see a folder with the name netcoreapp2.1, inside this folder, there will be a folder win7-x64, this is the folder where your exe file, log file and other items are residing.

Install the Service

To install our Asp.Net console application as a windows service, you can use the sc command. Open the command prompt with administrator privilege and run the preceding command.

And then you can start the service by running the command as “sc start BackupService”. This should start your service.

Giving Permission to the Folders

Sometimes you may get some permission issue in your service as it doesn’t have enough permission to read the files form the C drive. To overcome this, you should give enough permission to the user. You can do this by editing the security properties of those folders.

Do the same for the BackupZip folder as well.

Output

If you have run the service correctly, you should see a log file with the name backupclientlogfile.txt inside your win7-x64 folder. Once the service is run, the logs will be written as follows.

You can also check your Azure Storage account to check whether the files have uploaded correctly or not.

Conclusion

Wow!. Now we have learned,

• about Windows Service and Asp.Net Console Application
• about How to create a Windows Service using Asp.Net Core
• about How to use Quartz scheduler
• about How to use NLog in Windows Service
• about How to configure Quartz scheduler to use Dependency Injection
• about How to Zip a folder and save the file
• about How to upload a file to the Azure blob storage

Your turn. What do you think?

Thanks a lot for reading. Did I miss anything that you may think which is needed in this article? Could you find this post as useful? Kindly do not forget to share me your feedback.

Kindest Regards
Sibeesh Venu

Introduction

I have tried many things with Azure Functions and you should be able to see then here. Here in this post we are going to see how we can create a job which runs in a particular day and time to delete the blobs we have in a blob containers in our Azure Storage Account. If you want to know how you can upload to the blobs using Azure Function, consider reading my article here.

Prerequisites

• You should have a valid Azure Subscription
• You should have a running Azure Function
• If you are not sure about how to create an Azure Function App, this post may help you.
• You should have a valid Azure Storage Account

Using the code

Configure the Dependency Injection in Azure Function

As we are going to inject our dependency via constructor we need to configure the same by creating a Startup.cs class in our solution. Let’s do that first. To configure, make sure that you had installed the Nuget Package Microsoft.Azure.Functions.Extensions.

Now create a new class and name it as Startup.cs and write the code as preceding.

Here we are adding a singleton service for our IBlobService. Do not forget to inherit your Startup class from FunctionsStartup.

Write the Azure Function

As we have set up our Startup class, now let us create our Azure Function. Follow this post to see how you can create one. And you can write the code as preceding.

Here we are making the Function to run on every Monday at 4 AM using the CRON expression. Make sure to check my previous post to see more about the CRON expression.

Below are the blob container names I have in my Azure Blob Storage.

Make sure to use the small letters for your Azure Blob Container names, otherwise you will see an exception as mentioned here.

Creating the Blob Service

Now let us create a new Interface for our service.

And then create a service BlobService.

Here you can see that in the PerformTasks function we are getting the blob container reference and then get all the blobs using ListBlobsSegmentedAsync and then cast it as ICloudBlob so that we can easily delete the blobs.

Make sure to add the AzureWebJobsStorage in your local.settings.json file and in the Azure Function Configuration in the portal.

Conclusion

Wow!. Now we have learned,

• about Azure Function and setting up the same
• about Time Trigger in Azure Function
• about CRON expressions in Azure Function
• how to set up dependency injection in Azure Function
• how to delete Azure blobs from Azure Blob Containers using Azure Function

You can always ready my Azure articles here.

Your turn. What do you think?

Thanks a lot for reading. Did I miss anything that you may think which is needed in this article? Could you find this post as useful? Kindly do not forget to share me your feedback.

Kindest Regards
Sibeesh Venu

Introduction

Working with Azure Function is always fun and with the help of other Azure Services, it gets even better. Here in this article, we are going to see how we can use the TimerTrigger of Azure Function to upload some file to the Azure Blob storage daily. The basic idea is to have job running every data at a specific time to perform the upload. I believe, this is enough for the introduction, let us start implementing them.

Background

If you are new to Azure Functions, I recommend you to read some related articles from here. The possibilities that you can do with Azure Function is countless, and here we are going to discuss one of them. We live in a world everything is automated, nobody is interested in doing any manual works. Writing an automated Job which runs everyday at a given time is never been easier with the help of Azure Function.

Prerequisites

• You should have a valid Azure Subscription
• You should have a running Azure Function
• If you are not sure about how to create an Azure Function App, this post may help you.
• You should have a valid Azure Storage Account

Azure Function and Azure Blob

Get the Storage Connection String

By default a new key with the name AzureWebJobsStorage will be created when you create an Azure Function in your Visual Studio Azure Function App. If it doesn’t you should create this in your local.settings.json file by getting the connection string from your storage account.

To get the connection string, go to your storage account and click on Access Keys under Settings blade. And then copy any one of the connection string listed. You can use this value to create the AzureWebJobsStorage key under Values in the local.settings.json file. You should also create this in the Azure Function configuration in the Azure Portal, as this local.settings.json file is just for local and it will not be pushed to the Azure Function when you publish your Functions.

Creating an Azure Function in Function App

To create a new Azure Function in the Azure Function App, right click on your project and then click on Add and then select New Azure Function. In the coming pop up you should be given an option to name your Azure Function. Once that is done you can choose what trigger you want to use, in this case I will select TimeTrigger.

Once you click on Ok, a new Function will be created for you with the time interval of 5 minutes. The Time Trigger Function uses NCronTab library to intercept the CRON expression. This is the expression that you see under the Schedule option on the above image. A usual CRON express has the below fields in it.

If you see the code you can see that a function Run is been created for you, with the time interval you had set.

Now we can say our Azure Function to run everyday at 9.45 AM by changing the CRON expression.

You can learn more about the CRON expressions here.

Upload the Data to Azure Blob

Now that we have our Azure Function ready to be run everyday as we wanted. It is time to write some code to generate the data and upload it to the blob. Before we do that we should make sure that we have installed the required Nuget package Microsoft.Azure.Storage.Blob.

Now we can rewrite the complete Function code as below.

As you can see that, I am creating an instance of CloudStorageAccount and then create a blob client by calling the function CreateCloudBlobClient(). Once we get that, we get the blob container reference by using the function GetContainerReference() and then get the block blob and finally call the UploadFromStreamAsync() function to upload the stream data.

The line var stream = getTheImportedData.ConvertToStream(); is to get the stream data from an API, which returns a JSON and then I convert it to a stream by calling an extension method.

Here EnviornmentVariables.StorageConnectionString is part of a shared static class which returns the AzureWebJobsStorage we set in the local.settings.json file.

Output

Now let us run our function in local and test. You should see an output as preceding.

Conclusion

Wow!. Now we have learned,

• about Azure Function and setting up the same
• about Time Trigger in Azure Function
• about CRON expressions in Azure Function
• how to upload data to Azure blob using Azure Function

You can always ready my Azure articles here.

Your turn. What do you think?

Thanks a lot for reading. Did I miss anything that you may think which is needed in this article? Could you find this post as useful? Kindly do not forget to share me your feedback.

Kindest Regards
Sibeesh Venu