We decided we will be using RabbitMQ https://www.rabbitmq.com/ an open source MIT licensed widely used messaging queue solution that can easily be plugged into the .net core solution.
RabbitMQ is highly scalable.
A queue is file based and will require allocated storage in the solution.
To ensure we do not overload the database with requests when the queues run, we’ve decided the following.
Queuing messages will happen instantaneously in the API container.
Dequeuing messages will happen in a different container
In intervals ex. every 30 seconds and batched together when writing data to the database.
Format of the items names could be something like:
Internal.<cvr>.task
Internal.<cvr>.updateCPR
External.<cvr>.task
Configuration will be in the application settings and will be used in all queues.
They would containDequeue interval in seconds
Retry time in seconds
Number of Retry before giving up.
Queue location root
There will be a basic queue service to use in the API
There will be a queue for each unique type of functionality we need to handle in a separate dequeuing solution in a different container.
Below is shown an image of how the architecture would look and how queue messages would flow.
...