![]() “Virtual Private Snowflake” (VPS) is its highest-priced tier, and can run a dedicated isolated version of Snowflake. It is multi-tenant over shared resources in nature and requires you to move data out of your VPC and into the Snowflake cloud. U/eran-levy1 love to hear your comments on the pricing side, can you give your thoughts on my following question?Snowflake was one of the first decoupled storage and compute architectures, making it the first to have nearly unlimited compute scale and workload isolation, and horizontal user scalability. Love to hear your comments toward my situation. this post gave a lot of details about why BQ is superior to SF \_vs\_snowflake\_in\_2022\_on\_gcp/ PS: Here is a gentleman's comment favoring BQ (I felt his comments were very detail oriented, so I copy it here) so that we as a community could potentially help newbies like me to understand the full picture I have heard BQ pricing model is very simple and easy to reason and budget about (just know how many GB your query has scanned, that's it) - that's the biggest factor that made us think twice before fully adopt SF as our org is thin-margin and very frugal from budgeting approval perspective (but I have heard that team using SF could have more productivity and short ramp-up time - team efficiency is also $$ as we all know). there might be more stages between 1 and 2 for other data science usage, but not in foreseeable 1 year.weekly crunching of 1 above to get updated result for 2 above. ![]() small (not so huge) summarization table(s) in the final area to serve production query.huge raw data (~initially 30 TB, and will go to 500TB in foreseeable future) get ingested in the raw area.Right now, I am in a situation that we short-listed to BQ and SF. U/data-ai-nerd I'd love to follow up to get more details on this subject. Especially as the size of your data grows. It’s often cheap or even free to get data in, but very expensive to get data out. However, vendor lock-in with data is always a concern regardless of the data storage solution selected. And if vendor lock-in is a concern I’d recommend avoiding proprietary Snowflake solutions unless it offers significant benefits for your use case. There are also many third party and open source connectors for snowflake. However, you can still use the Snowflake connector with Spark just fine. For example, Snowflake does offer Snowpark, which is their Apache Spark equivalent. While Snowflake is trying to be more than a data warehouse and offering many proprietary solutions, you don’t have to use them. This also means that you can replicate databases to another cloud provider for security, backup, regulatory requirements, or even performance. Meaning that you can choose to host Snowflake on AWS, Azure, or Google Cloud. One big difference with Snowflake is that it is one of the few (only?) multi-cloud data warehouses. If you understand how you are charged then you can optimize. Cost depends a lot on usage patterns AND understanding in great detail how pricing works on your chosen platform. I don’t know much about BigQuery, but my understanding is that both data warehouses are very similar. So although you get VENDOR lock-in with Snowflake (less with Databricks due to much of the tooling being open source), you don't get CLOUD lock-in, which can be just as bad. ![]() Even if your company is already multi-cloud, SEAMLESS multi-cloud is VERY rare in my experience.ĭatabricks operates generally the same on all three platforms (GCP tends to get the short end of the stick), and Snowflake the same, though there are some minor differences. I'd say challenging but that's a huge understatement. But other times it's just not possible to orchestrate tools, and they generally won't work as well together, and your networking and security are going to be. orchestrate your BigQuery work from MWAA in AWS using code in Azure Devops. It's sometimes possible to be multi-cloud - e.g. Generally the proprietary tools of one stack are purposely integrated well with one another so that if you use one feature of AWS for example, you're incentivized to use others.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |