Find Jobs
Hire Freelancers

Database provider recommendation for scaling to larger size for Postgres DB.

$30-250 USD

Закрито
Опублікований over 7 years ago

$30-250 USD

Оплачується при отриманні
Database requirements recommendation We have an existing Postgres database and wish to upgrade it to a system that can enable fast, cost effective scaling much above what our system currently is. At present we have only a small database with about 1 million records running on shared hosting. Uptime: >99.995% uptime ~ 30 minutes per year A provider with a history of over this for 2 or more years is required so we can have confidence they will continue to achieve this uptime. Scaling We want a system that can scale up to 1 million new records per day, without having to change much except the monthly payment. Each record will have about 10 columns and 100 to 1000 chars, averaging 250 characters. This is about 250MB raw data per day (I don’t know what is normal regarding compression, it is text so will be able to be compressed) or a 25GB increase in 3 months. Now those records won’t need to be kept indefinitely, we expect >90% of those records would not need to last for longer than 3 months. For the long term records, we expect 100,000 new records per day needing to be kept for 2 years. This is about 100 million records cumulative after 2 years. So to summarize: 1 million new records per day of which 100,000 are kept for two years, 900,000 kept for 3 months. Possible further scaling: We are also interested in costs at 100M new records per day. For those records however they will most likely be in 1-2 tables and things like web usage logs. Like before only needing to be kept for 3 months. I have been told that that NoSQL is good for this sort of thing and to consider two databases, a SQL and NoSQL for this. I would prefer to just have one SQL one, however at stage could it make sense to split our system and use two different database types if any? Costs We want costs scale with database size, staying below $1K per month while we have a small database. Size: < 1GB Cost: <$1K per month Entries: 1 million new records, ~100MB size per day: Cost: <$10K per month preferred. Entries: 100M new records Cost: <$100K per month preferred. Backup Continuous backup with the ability to restore within 30 minutes for database sizes < 1GB. For example we want the ability to restore to points 30 mins, 12 hours, 1,2,3 days, 1 week previous etc. Cloning for testing With a few clicks we want to be able to make a clone database for staging/testing within 30 mins while the DB size < 1GB. Schema The schema is pretty simple, most data will be going into a single table, >95% in a few tables. Time to read/write 500ms per query is an acceptable time to retrieve data from the database for our users. I expect AWS or another cloud system would best meet all these requirements however open to surprises. Deliverables: Recommend a database provider taking all these requirements into consideration backed up by some evidence and experience in such things. To apply for the job, don’t make recommendations, just post evidence that your company has the expertise to do so.
ID проекту: 12268741

Про проект

3 пропозицій(-ї)
Дистанційний проект
Активність 7 yrs ago

Хочете заробити?

Переваги подання заявок на Freelancer

Вкажіть свій бюджет та терміни
Отримайте гроші за свою роботу
Опишіть свою пропозицію
Реєстрація та подання заявок у проекти є безкоштовними
3 freelancers are bidding on average $265 USD for this job
Аватарка користувача
Dear Client I am provide you worldwide fresh contacts with detail we chat more with sample USA Casino Players 700 USA Daily Phone Leads 100,000,000 USA Dating Leads 5,000,000 USA DEBT-RELIEF Leads 5,000,000 USA Diabetic Leads 15,000,000 USA Doctors 850,000 USA EDUCATION LEADS 100,000,000 USA Experian AOL 70,000,000 USA Experian Database with full details 140,000,000 USA Gmail Data with full details 190,000,000 USA INFOUSA Database of Business 1,195,130 USA Insurance Leads with full details 130,000,000 USA JigSaw Business Database 70,000,000 USA Job Seekers Leads with full details 20,000,000 USA Linkedin Only Emails 180,000,000 USA MORTGAGE Leads with full details 10,000,000 USA Pay Day Loans leads with full details 150,000,000 USA Payday Loan Long 100,000,000 USA PHONES_DNC_GENERAL_COREG 30,000,000 USA Photographers 100,000 USA Premium-Optin Leads with full details 200,000,000 USA Publisher Direct Leads with full details 15,000,000 USA Real Estate Agents 1,000,000 USA Real Estate Leads 5,000,000 USA SPORTS-ENTERTAINMENT Leads 10,000,000 USA STOCK-INVESTING 10,000,000 USA SWEEPSTAKES-FREE_OFFERS-SURVEY 10,000,000 USA CELL no 100,000,000 USA TRAVEL Leads 12,000,000 USA WEB_SERVICES 1,000,000 USA YAHOO Data with full details 100,000,000 USA Zoosk Dating with full details 50,000,000 USA Whios data web with full detail 20,000,000 etc thanks
$250 USD за 3 дні(-в)
0,0 (2 відгуки(-ів))
0,0
0,0
Аватарка користувача
Hi, You described your requirements quite clear to make it straight forward into implementation. I see this as a small Greenplum database in AWS - I think it will be the best combination of functionality, reliability and a cost of course. GP has partitioning for tables and external tables based on S3 which can be used for your historical data processing. I have an experience in building Greenplum clusters in AWS for our company data migration projects. GP in our case operates with several TB of data coming from on-premises Oracle databases by GoldenGate. I developed automation scripts and tools around (python scripts + Ansible roles and Jenkins jobs) to launch any size of data for production and for developers and test purposes. With AWS CLI (Boto3) and GP utils I built several maintenance scripts to simplify launch, terminate, migrate clusters which can be used for cloning database for some other needs. Ideally to speedup delivery of the project you should have AWS basic infrastructure up and running for your account. Performance tests should be prepared according to real business cases in advance and have to be prepared to run on PostgreSQL-like databases.
$244 USD за 5 дні(-в)
0,0 (0 відгуки(-ів))
0,0
0,0

Про клієнта

Прапор NEW ZEALAND
Christchruch, New Zealand
5,0
75
Спосіб оплати верифіковано
На сайті з квіт. 28, 2010

Верифікація клієнта

Дякуємо! Ми надіслали на вашу електронну пошту посилання для отримання безкоштовного кредиту.
Під час надсилання електронного листа сталася помилка. Будь ласка, спробуйте ще раз.
Зареєстрованих користувачів Загальна кількість опублікованих робіт
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Завантажуємо для перегляду
Дозвіл на визначення геолокації надано.
Ваш сеанс входу закінчився, і сеанс було закрито. Будь ласка, увійдіть знову.