Finding the best data engineering suitable for your needs isnt easy. With hundreds of choices can distract you. Knowing whats bad and whats good can be something of a minefield. In this article, weve done the hard work for you.
Best data engineering
1. Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems
Description
Data is at the center of many challenges in system design today. Difficult issues need to be figured out, such as scalability, consistency, reliability, efficiency, and maintainability. In addition, we have an overwhelming variety of tools, including relational databases, NoSQL datastores, stream or batch processors, and message brokers. What are the right choices for your application? How do you make sense of all these buzzwords?
In this practical and comprehensive guide, author Martin Kleppmann helps you navigate this diverse landscape by examining the pros and cons of various technologies for processing and storing data. Software keeps changing, but the fundamental principles remain the same. With this book, software engineers and architects will learn how to apply those ideas in practice, and how to make full use of data in modern applications.
- Peer under the hood of the systems you already use, and learn how to use and operate them more effectively
- Make informed decisions by identifying the strengths and weaknesses of different tools
- Navigate the trade-offs around consistency, scalability, fault tolerance, and complexity
- Understand the distributed systems research upon which modern databases are built
- Peek behind the scenes of major online services, and learn from their architectures
2. Feature Engineering for Machine Learning: Principles and Techniques for Data Scientists
Description
Feature engineering is a crucial step in the machine-learning pipeline, yet this topic is rarely examined on its own. With this practical book, youll learn techniques for extracting and transforming featuresthe numeric representations of raw datainto formats for machine-learning models. Each chapter guides you through a single data problem, such as how to represent text or image data. Together, these examples illustrate the main principles of feature engineering.
Rather than simply teach these principles, authors Alice Zheng and Amanda Casari focus on practical application with exercises throughout the book. The closing chapter brings everything together by tackling a real-world, structured dataset with several feature-engineering techniques. Python packages including numpy, Pandas, Scikit-learn, and Matplotlib are used in code examples.
Youll examine:
- Feature engineering for numeric data: filtering, binning, scaling, log transforms, and power transforms
- Natural text techniques: bag-of-words, n-grams, and phrase detection
- Frequency-based filtering and feature scaling for eliminating uninformative features
- Encoding techniques of categorical variables, including feature hashing and bin-counting
- Model-based feature engineering with principal component analysis
- The concept of model stacking, using k-means as a featurization technique
- Image feature extraction with manual and deep-learning techniques
3. Data Engineering
Description
If you found a rusty old lamp on the beach, and upon touching it a genie appeared and granted you three wishes, what would you wish for? If you were wishing for a successful application development effort, most likely you would wish for accurate and robust data models, comprehensive data flow diagrams, and an acute understanding of human behavior.The wish for well-designed conceptual and logical data models means the requirements are well-understood and that the design has been built with flexibility and extensibility leading to high application agility and low maintenance costs. The wish for detailed data flow diagrams means a concrete understanding of the business' value chain exists and is documented. The wish to understand how we think means excellent team dynamics while analyzing, designing, and building the application.
Why search the beaches for genie lamps when instead you can read this book? Learn the skills required for modeling, value chain analysis, and team dynamics by following the journey the author and son go through in establishing a profitable summer lemonade business. This business grew from season to season proportionately with his adoption of important engineering principles. All of the concepts and principles are explained in a novel format, so you will learn the important messages while enjoying the story that unfolds within these pages.
The story is about an old man who has spent his life designing data models and databases and his newly adopted son. Father and son have a 54 year age difference that produces a large generation gap. The father attempts to narrow the generation gap by having his nine-year-old son earn his entertainment money. The son must run a summer business that turns a lemon grove into profits so he can buy new computers and games. As the son struggles for profits, it becomes increasingly clear that dad's career in information technology can provide critical leverage in achieving success in business. The failures and successes of the son's business over the summers are a microcosm of the ups and downs of many enterprises as they struggle to manage information technology.
4. Database Reliability Engineering: Designing and Operating Resilient Database Systems
Feature
O Reilly MediaDescription
The infrastructure-as-code revolution in IT is also affecting database administration. With this practical book, developers, system administrators, and junior to mid-level DBAs will learn how the modern practice of site reliability engineering applies to the craft of database architecture and operations. Authors Laine Campbell and Charity Majors provide a framework for professionals looking to join the ranks of todays database reliability engineers (DBRE).
Youll begin by exploring core operational concepts that DBREs need to master. Then youll examine a wide range of database persistence options, including how to implement key technologies to provide resilient, scalable, and performant data storage and retrieval. With a firm foundation in database reliability engineering, youll be ready to dive into the architecture and operations of any modern database.
This book covers:
- Service-level requirements and risk management
- Building and evolving an architecture for operational visibility
- Infrastructure engineering and infrastructure management
- How to facilitate the release management process
- Data storage, indexing, and replication
- Identifying datastore characteristics and best use cases
- Datastore architectural components and data-driven architectures
5. Clean Architecture: A Craftsman's Guide to Software Structure and Design (Robert C. Martin Series)
Description
Practical Software Architecture Solutions from the Legendary Robert C. Martin (Uncle Bob)
By applying universal rules of software architecture, you can dramatically improve developer productivity throughout the life of any software system. Now, building upon the success of his best-selling books Clean Code and The Clean Coder, legendary software craftsman Robert C. Martin (Uncle Bob) reveals those rules and helps you apply them.
Martins Clean Architecture doesnt merely present options. Drawing on over a half-century of experience in software environments of every imaginable type, Martin tells you what choices to make and why they are critical to your success. As youve come to expect from Uncle Bob, this book is packed with direct, no-nonsense solutions for the real challenges youll facethe ones that will make or break your projects.
- Learn what software architects need to achieveand core disciplines and practices for achieving it
- Master essential software design principles for addressing function, component separation, and data management
- See how programming paradigms impose discipline by restricting what developers can do
- Understand whats critically important and whats merely a detail
- Implement optimal, high-level structures for web, database, thick-client, console, and embedded applications
- Define appropriate boundaries and layers, and organize components and services
- See why designs and architectures go wrong, and how to prevent (or fix) these failures
Clean Architecture is essential reading for every current or aspiring software architect, systems analyst, system designer, and software managerand for every programmer who must execute someone elses designs.
Register your product at informit.com/register for convenient access to downloads, updates, and/or corrections as they become available.
6. Python Data Science Handbook: Essential Tools for Working with Data
Feature
O'Reilly MediaDescription
For many researchers, Python is a first-class tool mainly because of its libraries for storing, manipulating, and gaining insight from data. Several resources exist for individual pieces of this data science stack, but only with the Python Data Science Handbook do you get them allIPython, NumPy, Pandas, Matplotlib, Scikit-Learn, and other related tools.
Working scientists and data crunchers familiar with reading and writing Python code will find this comprehensive desk reference ideal for tackling day-to-day issues: manipulating, transforming, and cleaning data; visualizing different types of data; and using data to build statistical or machine learning models. Quite simply, this is the must-have reference for scientific computing in Python.
With this handbook, youll learn how to use:
- IPython and Jupyter: provide computational environments for data scientists using Python
- NumPy: includes the ndarray for efficient storage and manipulation of dense data arrays in Python
- Pandas: features the DataFrame for efficient storage and manipulation of labeled/columnar data in Python
- Matplotlib: includes capabilities for a flexible range of data visualizations in Python
- Scikit-Learn: for efficient and clean Python implementations of the most important and established machine learning algorithms
7. Data Science on the Google Cloud Platform: Implementing End-to-End Real-Time Data Pipelines: From Ingest to Machine Learning
Description
Learn how easy it is to apply sophisticated statistical and machine learning methods to real-world problems when you build on top of the Google Cloud Platform (GCP). This hands-on guide shows developers entering the data science field how to implement an end-to-end data pipeline, using statistical and machine learning methods and tools on GCP. Through the course of the book, youll work through a sample business decision by employing a variety of data science approaches.
Follow along by implementing these statistical and machine learning solutions in your own project on GCP, and discover how this platform provides a transformative and more collaborative way of doing data science.
Youll learn how to:
- Automate and schedule data ingest, using an App Engine application
- Create and populate a dashboard in Google Data Studio
- Build a real-time analysis pipeline to carry out streaming analytics
- Conduct interactive data exploration with Google BigQuery
- Create a Bayesian model on a Cloud Dataproc cluster
- Build a logistic regression machine-learning model with Spark
- Compute time-aggregate features with a Cloud Dataflow pipeline
- Create a high-performing prediction model with TensorFlow
- Use your deployed model as a microservice you can access from both batch and real-time pipelines
