SQL coders horror stories

[Marcell1992]

Greetings SQL users.

I'm busy job hunting, and was looking to acquire a further education to advance my career, and data analysis caught my attention. Before I commit to it, I'd like to hear from fellow SQL users, what was one of the worst stories you can tell about the field?

I want to more or less get an idea of what's the worst one could expect and if it would be too overwhelming for me.

I did 1 free lesson so far on data analysis, and CSV seems pretty straight forward and I don't think it would be too complex to deal with. SQL also seem straight forward, but a lot more complex, and I'm sure in there somewhere lies a metaphorical right hook to my chin.

If it's of interest, I was a "data analyst" for about 4 years, I am now a "technical specialist", however these are more quasi titles of something vaguely related.
In these roles, I would analyze staff performance and draw up conclusions and training needs.
Setup or aid in setting up SOPs for my department and/or edit SOPs based on the aforementioned analysis
Internal Auditing, but again in a quasi way.
I enjoy the job, but absolutely hate the management

 

[Switch]

Just be aware for education (course content from tertiary education) that data analysis in the field of Big Data doesn't typically use sql, you tend to find nosql implementations more common with mongodb, analysis done using R or python.

 

[Marcell1992]

Just be aware for education that data analysis in the field of Big Data doesn't typically use sql, you tend to find nosql implementations more common with mongodb, analysis done using R or python.

Thanks. The job offers I've looked at that piqued my interest in the field all advertised SQL as a requirement. Those jobs will most definitely be filled by the time I am qualified, but it gives me an idea. So I was thinking of getting a hang of that first, before nosql. My friend that's been in the field for years, also recommend to learn SQL, and I'm talking like 15/10 recommend.

 

[Switch]

Thanks. The job offers I've looked at that piqued my interest in the field all advertised SQL as a requirement. Those jobs will most definitely be filled by the time I am qualified, but it gives me an idea. So I was thinking of getting a hang of that first, before nosql. My friend that's been in the field for years, also recommend to learn SQL, and I'm talking like 15/10 recommend.

You can learn sql, has relevance. Just from a subject perspective Big Data isn't oriented around sql, companies that use sql for it tend to be companies that have a large amount of legacy code and existing data stored within sql schema. Big Data [analysis] is more optimized with nosql implementations and the various analysis tools, so you are going to need to do more wrangling than usual if you want to use existing sql and translate it to take advantage of the nosql implementation. It's like you can still use table layouts to make websites, it's just that the web industry tends to use javascript and frameworks like react/angular.

 

[zakk]

You don't have to know it all but understanding when you would use a type of database is a good foundation. These would be a great start.

Document store
Full text search
Relational database
Wide-column store
Triple store

It all comes down to understanding a data model and applying a said query language to it. That said SQL is far from dead and can be implemented on quite a few noSQL tech's as well although you need to understand the data model to find constraints.

I mostly use Apache Spark SQL that is querying a large (billions of records) wide-columnar store that is a noSQL database. However I have to replicate my data 5x (w/ 5x replication I use around 75GB per billion records) to achieve all the query power I require. So there are trade offs and this makes noSQL somewhat of a more specialized field I'd say. It's 'newer' and people are less aware of how to model data at scale.

My use case is definitely not the only and there are insane solutions for running SQL on multiple noSQL datastores, Apache Phoenix being one of my favorite. Takes a SQL statement and convert it into HBase table scans.

Anyway Gremlin is also very good to know when you touch Triple stores but yes you really can't go wrong with SQL.

SQL is extremely powerful and even though I mainly work with 'Big Data' there's always still a relational DB doing something in the stack - and a lot of other SQL jobs in Spark.

So learn it and understand when to use which technology I think it's an awesome field but I love data.

What do I do? I profile people's movement data into events to assist business' in choosing market strategies (where to play and how to win) especially in areas where transactional data is not present.

 

[HButton]

Update table forgetting the where statement that should be used with it...

 

[Marcell1992]

Update table forgetting the where statement that should be used with it...

lol is that the biggest nightmare you've encountered? Sounds hopefully like a redo thing only

 

[HButton]

lol is that the biggest nightmare you've encountered? Sounds hopefully like a redo thing only

Sounds simple, but updating a status for monitoring services to pickup transactions to process and create duplicates for external systems for processing and we talking 100k + records

Also yes I am that good for this be my only mistake.

And yes you made me cry having to go into detail. ~ Me from the shower

 

[Dom]

Don't even really work with SQL that much - trapped error @Moemfie_ZA

 

[∀Z‾soʇɐuɐɥ┴]

lol is that the biggest nightmare you've encountered? Sounds hopefully like a redo thing only

Tbf this is generally the worst way you can cock up any update/delete statement in SQL... especially if you check it on a handful of records that should have been updated, but not on ones that should not have been updated then you pick it up a few days/weeks later.

With SQLServer there is a way to create a recycle bin for deleted records, but not so much on FlameRobin, however even on SQLServer it's a pain in the ass from what I have seen.

 

[Marcell1992]

Tbf this is generally the worst way you can cock up any update/delete statement in SQL... especially if you check it on a handful of records that should have been updated, but not on ones that should not have been updated then you pick it up a few days/weeks later.

That does sound tedious with more detail, because I'm sure making backups as you go won't have any effect as your newly added records would be gone anyway. So how would one then need to resolve this and also pull records that fell through the filters/parameters?

 

[∀Z‾soʇɐuɐɥ┴]

That does sound tedious with more detail, because I'm sure making backups as you go won't have any effect as your newly added records would be gone anyway. So how would one then need to resolve this and also pull records that fell through the filters/parameters?

don't add commit; to the end of your statements/procedures when testing and check the number of records updated/deleted to make sure it matches the nr of records you want to update/delete...