Case Study - Medical Data Indexer
Design and implementation of a Big Data Backend System capable of storing Medical Insurance Plan documents. The system is highly scalable and capable of indexing JSON documents for extremely fast searching.
- Industry
- Insurance
- Year
- Service
- Big Data
Challenge
The project required the design and implementation of a Big Data Backend System capable of storing Medical Insurance Plan documents. The backend system needed to be highly scalable and capable of indexing JSON documents for extremely fast searching.
The specific requirements were as follows:
- Design a system capable of storing a large number of medical insurance plan documents.
- The system should be highly scalable to handle the growing number of documents.
- Implement a mechanism to index the documents for extremely fast searching.
- Ensure the system is containerized for easy deployment and scaling.
- Implement OAuth 2.0 Authentication to secure the system.
- Use message queues to feed data simultaneously to multiple databases.
- Implement measures to alleviate potential server bottlenecks.
- Ensure the system is decoupled to increase concurrency.
Solution
The solution involved the following key implementations:
- Designed and formulated a containerized system capable of storing a large number of medical insurance plan documents.
- The system was developed using Go language, ensuring efficient performance.
- Implemented OAuth 2.0 Authentication to secure the system and protect sensitive data.
- Integrated Elasticsearch for quick lookup of insurance plans, enhancing the system's search capabilities. Used Parent Child Mapping to store nested documents.
- Leveraged RabbitMQ exchange-based message queues to handle data feeds to multiple databases including Redis and ElasticSearch.
- Implemented measures to alleviate potential server bottlenecks, ensuring smooth system operation.
- Achieved system decoupling and increased concurrency, enhancing system performance and scalability.
Technologies
ElasticSearch
Redis
RabbitMQ
Go
OAuth
Docker
Git
