data infrastructure
what you and others generically refer to as infrastructure or the data center may, in fact, be the data infrastructure .
cisco prime infrastructure 3. 2 administrator guide - data collection and background tasks [ cisco prime infrastructure ] - cisco
cisco prime infrastructure 3. 0 administrator guide - managing data and collection retention [ cisco prime infrastructure ] - cisco
data is becoming essential to daily life, and having a proper data infrastructure helps in collecting insights from data in a better way. a data infrastructure is a collection of data assets, the bodies that maintain them and guides that explain how to use the collected data. a data infrastructure is the proper amalgamation of organization, technology and processes .
system health records: this includes most data shown on prime infrastructure administrator dashboards .
the edison project: building the data scientist profession and coordinating education and training for data and e - infrastructure .
privacy of data is an important aspect, and thus the data assets in a data infrastructure could either be in the open part or in the shared form. data can create maximum value if it has an open data infrastructure. however, if the contents are critical, data protection is required .
– system health records: this includes most data shown on prime infrastructure administrator dashboards .
for device and interface performance data, prime infrastructure uses the values specified in the fields under device health data retain periods .
the e - infrastructure advisory group identified a pressing need for guidance and dialogue on “big data
.
prime infrastructure supports data migration from cisco prime lan management solution (lms) version 4. 2. 5 on all platforms. the following lms data can be imported into prime infrastructure using the car cli :
exit your cli session, log back in to the prime infrastructure user interface, and verify that your lms data was imported properly. the following table shows where to look in prime infrastructure for the imported lms data .
whether you realize it or not, you may already be using, rely upon, affiliated with, support or otherwise involved with data infrastructures. granted what you or others generically refer to as infrastructure or the data center may, in fact, be the data infrastructure .
step 5 exit your cli session, log back in to the prime infrastructure user interface, and verify that your lms data was imported properly. the following table shows where to look in prime infrastructure for the imported lms data .
in order to get going with big data and turn it into insights and business value, it’s likely you’ll need to make investments in the following key infrastructure elements: data collection, data storage, data analysis, and data visualization / output. let’s look at each area in turn .
prime infrastructure stores all data it receives about network elements and protocols, including any duplicate data that it may receive from multiple sources. when you specify authoritative data sources, only the data from the specified sources is displayed when you view a particular location or site .
to address the above challenges, it became clear that 2017 would be the year we’d double down on our data infrastructure investment .
administrators can use the “other data retention criteria” section of the data retention page to configure retention periods for specific prime infrastructure database tables. you specify the retention period using the following attributes :
non - aggregated historical data—numeric data that cannot be gathered as a whole or aggregated. client association history is one example of non - aggregated historical data .
lets you schedule automatic prime infrastructure server backups. the backups created are application backups .
in the course of a few months, we established more robust and stable data infrastructure. to complete the entire picture, we needed :
machine learning infrastructure: many products at airbnb rely on machine learning (ml) to achieve their goals. a common infrastructure for ml can save significant development time for the company .
check box to allow prime infrastructure to enable syslog and trap notifications on newly added devices .
step 2 in repository path, specify the report repository path on the prime infrastructure server .
aggregated historical data—numeric data that can be gathered as a whole and summarized as minimums, maximums, or averages. client count is one example of aggregated historical data .
1. non - aggregated historical data—numeric data that cannot be gathered as a whole or aggregated. client association history is one example of non - aggregated historical data .
real - time streaming infrastructure: to enable teams to move quickly, getting accurate data with minimal delay is a core focus in di. currently, we are building out spark - based infrastructure to allow for easy development of streaming applications .
data on client session histories. you can also specify the maximum number of session entries to keep, specified as rows in the prime infrastructure database .
open a cli session with the prime infrastructure server (see how to connect via cli) .
two related terms deserve mention. critical infrastructure refers to those infrastructure assets critical to the functioning of society. while this traditionally has referred to infrastructure dealing with necessities such as the water supply, electricity, transportation, and food supply, the federal government now includes information technology in critical infrastructure, both in terms of the organizations that supply it and the organizations that use it. 1
prime infrastructure performs scheduled data collection jobs on a regular basis. you can change each job’s schedule, pause or resume it, or execute it immediately .
repeat this step as needed for each location and application data type for which you want to specify authoritative data source .
2. aggregated historical data—numeric data that can be gathered as a whole and summarized as minimums, maximums, or averages. client count is one example of aggregated historical data .
all data collection tasks (and data purging tasks) are controlled from the jobs dashboard. see manage jobs using the jobs dashboard. data collection jobs are listed under system jobs .
disabling or limiting these background data collection jobs can have a direct impact on how you use prime infrastructure, especially for reporting. to help you consider these impacts, take note of the reports this data is used in .
check box to allow prime infrastructure to collect inventory when it receives a syslog event for a device .
create linkdown event—select this check box to have prime infrastructure generate alarms for both access and trunk ports .
department of homeland security, national infrastructure protection plan: partnering to enhance protection and resiliency, 2009 .
you can read about how we' re using performance analysis and data science in government on the cross - government data blog .
the data deduplication page allows you to specify one or more authoritative data sources at a specific location. for example, if you have a network analysis module (nam) at a branch office as well as netflow data that is sent from the same branch, you can choose to have prime infrastructure display only the nam or the netflow data for that location .
prime infrastructure performs scheduled data collection jobs in the background on a regular basis. you can change each job’s schedule, pause or resume it, or execute it immediately .
step 2 select the enable data deduplication checkbox and click apply. the data deduplication page displays the list of your defined location groups .
establish a coordination framework between data infrastructures and related projects, as well as key industry partners with respect to the data scientist profession .
this source data containing user activity event data and dimensional snapshots is then sent into the gold cluster where we store the data and begin running our extraction, transform and load jobs. in this step, we apply business logic, make summary tables, and perform data quality checks .
if the device (s) you select has configuration changes and prime infrastructure detects that the change is one of the specified commands in the exclude list, prime infrastructure will not create an archived version of the configuration with this change .
from the drop - down list, select the data sources you want to specify as authoritative for that location and application data type. then click
there are different infrastructure layers about information technology (it) from higher - level business systems (e. g. business infrastructure), along with applications and environments for those who support and create them (e. g. information infrastructures). then there are the data infrastructures along with lower level physical facilities or real data centers also known as habitats for technology. the physical facilities can, in turn, rely on external electrical power, network communications, and other infrastructure resources or services .
. if it can identify them, prime infrastructure will fill in the address of an authoritative source in the list box under the column listing sources for each of the classes of application data .
the inventory page allows you to specify if prime infrastructure must collect inventory when a syslog event is received for a device .
with backup device configuration enabled, prime infrastructure automatically backs up all device running and startup configurations before deploying new configuration templates .
interval—enter the interval, in days, between each automatic prime infrastructure backup. the valid range is 1 to 7 days .
step 2 open a cli session with the prime infrastructure server (see “connecting via cli” in related topics, below) .
many research questions involve the analysis and interpretation of big data - sets. handling and analysing “big data” is the largest software challenge facing research consortia .
changes you make on the data retention page determine the information that is displayed in the web gui. you can open the data retention page by choosing
for the foundational layer of our data infrastructure, we required a tool to “stitch” all our production data together. the solution could have been a consolidated database or a federated querying engine with access to various sources of production data. given the nature of our initial needs (i. e. analytical data could be latent), we agreed to first take the traditional etl approach and focus on creating stable batch snapshots of our data. in the end, we chose presto as our query engine to provide federated access to numerous data sources, as we saw several advantages :
thus, in 2017, we started a significant push to building out our data infrastructure. what that meant depended on the demand we received both internally and externally, which we identified as the following :
the framework laid out here provides background to enhance understanding of the types of infrastructure projects, present capabilities, analyze benefits and risks, develop a cost model, and, by comparing alternatives, develop the value proposition for infrastructure projects for presentation to decision makers .
creative engineers and data scientists building a world where you can belong anywhere. urltoken
step 1 choose administration > settings > system settings > general > data retention .
data collection jobs run on a regular default schedule, as described in the related topic “about data collection jobs”. you can re - schedule them as needed .
repeat—enter the number of times that you want the data collection to be repeated .
disabling or limiting these system jobs can have a direct impact on how you use prime infrastructure, especially for reporting. to help you consider these impacts, take note of the reports this data is used in .
personal data is data that can be used to identify an individual. this could be their name, address, date or country of birth, bank details or national insurance number. the way personal data is used and stored is protected by …
c. from the drop - down list, select the data sources you want to specify as authoritative for that location and application data type. then click ok .
involve multiple stakeholders (including projects) in discussions about the competences and skills required for data infrastructures taking the proposed edison data science competence framework as a starting point .
with a little technical knowledge, you can set many of these systems up yourself, or you can partner with a data company to set up the systems and capture the data on your behalf. accessing external data sources, such as social media sites, may require little or no infrastructure changes on your part, since you’re accessing data that someone else is capturing and managing. if you’ve got a computer and an internet connection, you’re pretty much good to go .
however, let’s get back to our focus which will be in and around the data infrastructure that spans from on - premise to cloud. as a refresh or to provide context, data infrastructures are the collection of hardware and software, from legacy physical to software - defined virtual, containers, cloud .
the data infrastructure team has done a lot of work lately to improve the way we talk about our products and services. one of the ways we’ve done this is by defining the terms we use to describe the things …
let’s wrap up for now, and put it all back together in that data infrastructures are what exist in data centers and physical facilities. data infrastructures can be small all in one hyper - converged infrastructures (hci) or converged appliances in smaller environments .
: specifies the maximum data retention period in hours for all records in the database .
you can pause any scheduled data collection job, and resume it if already paused .
the following topics explain how to achieve these goals and perform related data management tasks .
you can also specify data retention for individual database tables, using maximum age and record attributes. for details, see “specifying data retention by database table” in related topics .
establishing a data refresh schedule that’s consistent with business needs, timing, and cycles .
of course, there will be the buzzword bingo topics such as aws, ai, analytics and automation, big data (and little data) along with bleached data (e. g. wipe or secure erase). there are also containers and cloud, data protection and data footprint reduction (dfr), energy and economics as well as efficiency (utilization or space savings) along with effectiveness (productivity and performance) .
the teams working on data infrastructure at gds have just completed three discoveries. we’ve walked away with some great products, but also some good learning about what does and doesn’t work when doing discovery. scope the brief discovery is …
when you choose administration > settings > system settings > general > data retention, you can modify the retention periods for performance data under performance data retain periods. the performance retention values you specify determine the information that is displayed in performance reports and performance dashboards .
step 3 select the enable syslog and traps on device check box to allow prime infrastructure to enable syslog and trap notifications on newly added devices .
although infrastructure projects aren’t intended to change user functionality, user productivity often decreases during the learning curve as users get used to the new elements .
limiting the amount of data retained can help improve performance and disk storage characteristics. however, for the best interactive graph data views, cisco recommends that keep the default values .
step 2 select the enable event based inventory collection check box to allow prime infrastructure to collect inventory when it receives a syslog event for a device .
data workflow management: started at airbnb, airflow is a system that enables users to schedule data - related workflows with a code - as - configuration model and web front end .
select the category of data collection job you want to run (e. g. ,
step 2 expand the data category for which you want to specify retention - period values .
this is where the data arrives at your company. it includes everything from your sales records, customer database, feedback, social media channels, marketing lists, email archives and any data gleaned from monitoring or measuring aspects of your operations. you may already have the data you need, but chances are you need to source some or all of the data required .
during the gartner data center, infrastructure & operations management conference in las vegas, david cappuccio, vice president and distinguished analyst at gartner, says that these technology trends that impact i & o fall under three areas — strategic, tactical and organizational .
likewise, data infrastructures can be rack - scale in larger environments all the way up to larger enterprise and cloud service providers. the key is that data infrastructures exist to enable, protect, preserve, secure and serve applications that transform data into information. combined information infrastructures and data infrastructures along with their underlying infrastructures as well as applications they support can be considered information factories .
generally speaking, people tend to refer to infrastructure as those things that support what they are doing at work, at home, or in other aspects of their lives. for example, the roads and bridges that carry you over rivers or valleys when traveling in a vehicle are referred to as infrastructure. this is also the situation with it systems and services where, depending on where you sit or use various services, anything below what you do may be considered infrastructure .
justifying an infrastructure project means presenting the value proposition. although cost reduction is often the justification for infrastructure investments, other goals are also involved, such as improved capability or risk reduction. infrastructure projects will compete with all the other demands on the it budget, however, and sometimes suffer in comparisons because benefits and risks are thought to be difficult to explain or quantify. a value analysis can be helpful in this regard and need not be costly or difficult to develop .
performance: this includes assurance data such a traffic statistics, application metrics, and voice metrics .
page to define aggregated data retention periods. aggregation types include hourly, daily, and weekly .
as presto handles all data manipulation in memory, there is limited “temporary storage” on disk needed .
presto connector. it’s easy to setup connection to presto. since we have already consolidated all data storages with presto + hive, connection with presto means we can query any data with airflow .
validating the data accuracy and consistency during the extract / transform / load (etl) process .
planning for the user training and support needed to learn how to use tools and access data .
prime infrastructure provides configuration archive support for devices running cisco wlc software. the configuration archive includes only startup configurations. the running configurations are excluded from configuration archive .
the answers to these questions can help an organization decide what and how much maintenance is appropriate, which provides the basis for a cost model for infrastructure maintenance .
save as prime infrastructure events routine client association and disassociation traps and syslogs. this option is disabled by default, to avoid prime infrastructure performance problems on large networks during periods (such as network setup) when these kinds of traps and syslogs may be numerous. you may want to enable this option at all other times .
we leverage existing open source technologies like kafka, hadoop, hive, presto, spark, and also write our own. many projects listed below have been open sourced. as a member of our team you would spend time designing and growing our existing infrastructure, democratizing data access at the company, and promoting the correct use of data and analytics at the company .
select the category of data collection job you want to re - schedule (e. g. ,
select the category of data collection job you want to pause or resume (e. g. ,
– performance: this includes assurance data such a traffic statistics, application metrics, and voice metrics .
this is where you keep your data once it is gathered from your sources. as the volume of data generated and stored by companies has exploded, sophisticated but accessible systems and tools have been developed to help with this task. the main storage options include: a traditional data warehouse; a data lake; a distributed / cloud - based storage system; and your company server or a computer hard disk .
then there are fast software, servers, storage and networks including flash ssd, google, health check tools, intel and iot / iod data infrastructure demands. needless to say there are many other related items as well as industry activities we will have opportunities to discuss .
step 3 to automatically detect authoritative sources at all locations, click auto - detect. if it can identify them, prime infrastructure will fill in the address of an authoritative source in the list box under the column listing sources for each of the classes of application data .
enabled, prime infrastructure automatically rolls back each device to its last archived running and startup configurations when any attempt to deploy a new configuration template to the device has failed .
joining across stores was difficult to impossible. since some data resided in mysql and other data resided in mongodb, complex reports often required a federated query, something that wasn’t easily manageable given the above architecture .
other promising opportunities for the use of linked open data are related to temporal information and information about geographic place - based data. i am excited to report on two efforts imls is supporting in these areas .
checkr is a cloud - based company, so to no surprise, a lot of our data is stored in the cloud, specifically on aws. with that, we wanted a cost effective way to store large chunks of data for data analysis and s3 was the obvious choice. while many traditional data warehouse solutions use distributed file systems such as hdfs, our presto + hive approach integrated particularly well with s3, because
never miss a story from airbnb engineering & data science, when you sign up for medium. learn more
snmp - polled device data such as device reachability, and utilization for cpu, memory, and interfaces .
schedules data collection based on specified command - line interface (cli) commands at a configured time interval .
planning for a user - friendly, powerful desktop query tool for users to access data without it assistance .
look to the open source world: there are a lot of good resources for data infrastructure in the open source community and we try to adopt those systems. furthermore, if we build something useful ourselves and it is feasible to give it back to the community, we reciprocate .
time of day—enter the time of the day that you want prime infrastructure server backups to be taken. use 24 - hour format (for example, 13: 49) .
save all 802. 1x and 802. 11 client authentication - failure traps as prime infrastructure events. this option is disabled by default, to avoid prime infrastructure performance problems on large networks during periods (such as network setup) when these kinds of traps and syslogs may be numerous. you may want to enable this option if your network is stable .
a data warehouse is a solution that brings together information from diverse sources and puts it in a format that stakeholders can easily access when making complex business decisions. a data warehouse supports a company’s tactical and strategic goals .
age (in hours): specifies the maximum data retention period in hours for all records in the database .
if you specify these settings, all data displayed in performance reports and performance dashboards data will be for the previous 7 days only. when you generate a performance report (for example, reports > reports > report launch pad > device > device health), even if you select a reporting period longer than the last 7 days, the report contains data from the last 7 days only because that is all the data you’ve selected to retain .
data warehouses are useful for trend analysis, forecasting, competitive analysis, and targeted market research. data is often summarized by specific subject area, function, department, geographic region, time period, or all of these .
the maximum number of thread pools to devote to the archive operation. increasing the default can be helpful with prime infrastructure performance during archiving of changes involving more than 1, 000 devices .
with rollback configuration enabled, prime infrastructure automatically rolls back each device to its last archived running and startup configurations when any attempt to deploy a new configuration template to the device has failed .
as organizations strive to align it and operational technology to drive digital business innovations, infrastructure & operations (i & o) leaders should focus on 10 key technology trends to support these initiatives .
on the registers team, we are building a product that provides teams with data that is good enough to build services on. registers are defined by certain characteristics such as being live data and being the only authoritative list for a …
lets you schedule data collection based on specified command - line interface (cli) commands at a configured time interval .
interval—enter the interval, in days, between each device data collection. the valid range is 1 to 360 days .
the internet of things (iot) will change how future data centers are designed and managed and how they evolve as massive volumes of devices stream data, constantly or periodically, to enterprises, government departments and agencies around the world. i & o should use an iot architect who looks at the long term strategy for both iot and the data center .
last year the data leaders network proposed that education should be one of the five areas we should focus on in developing the first tranche of registers. school - related data in particular is widely used, by a large and varied number …
device - related historical information. trend data is gathered as a whole and summarized as minimums, maximums, or averages .
in this blog post, you’ll see an overview of how we established a stable data infrastructure to fuel our growth in the matter of a few months. in the near future, we’ll release more blog posts to highlight key components of this architecture. we hope our insights can be useful for companies facing similar challenges at scale .
interactive dimensional analysis: data scientists have a strong need to query data and compute aggregates on various dimensional cuts. to address this, we are building a query tool based on druid to allow users to interactively slice - and - dice large datasets .
if you specify these settings, all data displayed in performance reports and on performance dashboards will be for the previous 7 days only. when you generate a performance report, even if you select a reporting period longer than the last 7 days, the report will contain data from the last 7 days only (because that is all of the data you selected to retain) .
at airbnb we promote a data informed culture and use data as a key input for making decisions. tracking metrics, validating hypotheses through experimentation, building machine learning models, and mining for deep business insights are all critical to our moving fast and moving smart .
administrators can choose to have device configurations backed up or rolled back whenever prime infrastructure users deploy new device configuration templates. they can also control how cisco wlc configurations are archived, as explained in the following related topics .
the following table describes the background tasks prime infrastructure performs. you can manage how and when they are performed by choosing administration > settings > system settings > background tasks, then clicking the hypertext link for that task .
ribes, david, and thomas finholt .
the long now of technology infrastructure: articulating tensions in development .
journal of the association for information systems 10, no. 5 (may 2009): 374–398 .
objective of the workshop the workshop will allow edison to take stock of the main initiatives and projects in the digital infrastructure ecosystem and map their contributions to the definition of an ambitious european framework for developing skills and competences .
unlock the potential of your data. how well are you harnessing information to improve business outcomes? a new cio playbook will help .
edison is a 2 - year project (started september 2015) with the purpose of accelerating the creation of the data science profession .
acknowledgements: checkr data infrastructure was built in collaboration with kurt ruppel, stewart park, grace lee, kevin woods and the rest of the customer report and money team. kurt ruppel, and michelle berry gave valuable edits to this post. additionally, many engineers and product managers across the company provided feedback. huge appreciation to everyone involved in the project !
with infrastructure projects, the changes often affect stakeholders, external customers, or suppliers. business analysts are involved to manage requirements and expectations of these changes among all project stakeholders. here are some things to keep in mind :
data sets are being constantly generated from experiments, computer simulation / modelling, the internet (including social media), and mobile devices .
important issues need to be addressed on ownership, curation and management, analysis and interpretation, standardisation and usability, anonymisation and data protection .
data on disassociated clients. the default is seven days, and this applies irrespective of whether the clients will ever attempt to associate again .
similarly, if you view a performance dashboard (for example, dashboard > overview > general > service assurance) and select a time frame longer than one week, the dashboard contains data from the last 7 days only because that is all the data you’ve selected to retain .
as it evolves to adopt these trends, some new positions will be required within the ranks of infrastructure and operations. first and foremost will be the it cloud broker, responsible for monitoring / management of multiple cloud service providers .
the section is categorized into multiple subsections. each subsection list each database table name, along with the current age and max records used to determine whether an individual record in the table will be retained or discarded. the page also lists the table age attribute used to compute the age of the data in the table. the optical devices category is not applicable for prime infrastructure .
the retention periods for these aggregation types are defined as default, minimum, and maximum (see the table below). use the administration > settings > system settings > general > data retention page to define aggregated data retention periods. aggregation types include hourly, daily, and weekly .
since we configured a large memory footprint for our presto cluster, disk io to s3 is limited as a large amount of data is cached .
it’s worth noting that there are also limitations with this approach, as the presto cluster needs to have enough memory to hold all data in - memory for any given query. while there are some possible optimizations available — including splitting up data into separate queries -, those approaches can be tedious to implement. by keeping data on s3, we achieved high reliability and were also able to keep multiple snapshots with relatively low cost .
data center interconnection fabric is poised to deliver on the promise of the data center as software - defined, dynamic and distributed. the ability to monitor, manage and distribute workloads dynamically, or to rapidly provision lan and wan services through an api, opens up a range of possibilities .
it leaders need to create a data center as a service (dcaas) model, where the role of it and the data center is to deliver the right service, at the right pace, from the right provider, at the right price. it becomes a broker of services .
next will be the iot architect, tasked with understanding the potential impact of multiple iot systems on data centers. this architect will also be working with business units to insure their closed loop iot solutions are either compatible with the central iot architecture or that common protocols and data structures are used .
global digital data expanded from 130 exabytes in 2005 to 1, 227 exabytes in 2010 with a predicted rise to 7, 910 exabytes by 2015 .
identifying what information the data warehouse must contain, identifying who should have access to it, and making sure users have the right level of access .
by storing all our hdfs data in mounted ebs (elastic block storage) volumes, we were sending lots of data over the public amazon ec2 network in order to run queries. hadoop was built for commodity hardware and expects local reads and writes on spinning disks, so this was a design mismatch .
step 2 select the category of data collection job you want to run (e. g. , inventory, wireless poller, or wireless system) .
to be clear, many companies use mesos to great effect and implement novel solutions to administer large sets of machines running important infrastructure. but, our small team decided running a more standard, ubiquitous deployment would reduce the time we spent on operations and debugging .
as mentioned in the earlier section, one motivation for building this infrastructure was to provide analytics to our customers. we’re lucky enough to be in a world with many mature analytics providers. we picked chartio as our analytics provider, primarily for the following reasons :
with all the etl pieces in place, we need to open up our dataset for data scientists and analysts, so we can extract values out of it .
you can also tell prime infrastructure to ignore for archive purposes any change that involves specified commands on devices of a given family, type, or model. this is useful when you want to ignore insignificant or routine changes in a few parameters on one or many devices .
device health: this includes snmp polled data for wired and wireless devices, such as device availability, and cpu, memory, and interface utilization, and qos .
step 2 select the category of data collection job you want to re - schedule (e. g. , inventory, wireless poller, or wireless system) .
step 2 select the category of data collection job you want to pause or resume (e. g. , inventory, wireless poller, or wireless system) .
infrastructure projects are internal technical upgrades that impact systems, hardware, platforms, or tools in order to improve the technology that supports the business and the information technology (it) efforts. typically, these projects are called it projects because they’re driven and sponsored by it departments .
. by default, the retention period for all non - aggregated historical data is 31 days or 1 million records. this retention period can be increased to 365 days .
– device health: this includes snmp polled data for wired and wireless devices, such as device availability, and cpu, memory, and interface utilization, and qos .
we intentionally skipped some other popular data infrastructure solutions, such as hadoop or redshift. hadoop requires a high level of maintenance, and as we currently aren’t concerned about operating at petabyte scale, it was a non - starter. we also elected presto over redshift because a) we’d identified it as a potentially expensive solution for our use - case and b) we didn’t want to maintain and update our schema (covered in the next section) .
digital data can be stored, shared, searched, combined and duplicated with extraordinary speed at comparatively low cost. importantly, it is accompanied by large amounts of highly descriptive metadata .
you can define a retention period (and other settings) for each non - aggregated data collection task. for example, you can define the retention period for client association history in
stranded capacity – things that are paid for, but not really used – can be found both in on - premise data centers and in the cloud. it leaders should learn to focus not just on uptime and availability, but also on capacity, utilization and density. fixing this can extend the life of an existing data center and reduce operating expenditures from providers .
big thanks to the team of engineers who built the original foundation of data infrastructure at airbnb and to the folks who have been steadily working to improve and stabilize the systems. i was happy to write this blog post but the real credit goes to nick barrow - williams, greg brandt, dan davydov, wensheng hua, swaroop jagadish, andy kramolisch, kevin long, jingwei lu, will moss, krishna puttaswamy, liyin tang, paul yang, hongbo zeng, and jason zhang .
this year we undertook a significant migration to go from a poorly architected set of clusters called “pinky and brain” and onto the “gold and silver” system described above. to set some context for scale, two years ago we moved from amazon emr onto a set of ec2 instances running hdfs with 300 terabytes of data. today, we have two separate hdfs clusters with 11 petabytes of data and we also store multiple petabytes of data in s3 on top of that. with that background, here were the major problem areas and what we did to resolve them :
in addition to the steps below, you can run a job immediately by rescheduling it and selecting the time to execute as “now” (see “scheduling data collection jobs” in related topics) .
ratings can be established in various ways, ranging from subjective opinions of people knowledgeable about the system to detailed technical analyses performed by outside consultants. once individual measures have been assessed, the infrastructure system or component can be given an overall rating or
health check score
depending on the decisions to be made :
leave some headroom: we oversubscribe resources to our clusters in order to foster a culture of unbounded exploration. it is easy for infrastructure teams to get wrapped up in the excitement of maximizing resources too early, but our hypothesis is that a single new business opportunity found in the warehouse will more than offset those extra machines .
cisco recommends you do not change the retention periods for trend, device health, system health, and performance data because the default settings are optimized to get the most helpful information from interactive graphs .
click the drop - down list box under the class of application data for which you want to specify an authoritative source (for example: click in the list box under “application response time”) .
you can define a retention period (and other settings) for each non - aggregated data collection task. for example, you can define the retention period for client association history in administration > settings > system settings > client. by default, the retention period for all non - aggregated historical data is 31 days or 1 million records. this retention period can be increased to 365 days .
in the beginning, we implemented some quick and dirty tools to enable infrequent querying of data. our solution was by no means scalable, but it was sufficient to answer some key business questions .
some early airbnb engineers had a keen interest in a piece of infrastructure called mesos, which sets out to deploy a single configuration across many servers. we built a single cluster of c3. 8xlarge machines in aws each backed by 3tb of ebs and ran all hadoop, hive, presto, chronos, and marathon on mesos .
data are organized around the concept of project with the project id being the key linking the project to related operational datasets such as contract awards, loans / credits / grants, trust funds, etc .
together these six projects illustrate how libraries are working together to develop strategies to leverage linked open data technology as a means to better integrate their work to provide content and services to users across the country .
we had been running a federated hdfs cluster with pinky and brain where the data was held in shared physical block pools but the sets of mappers and reducers were isolated per logical cluster. this led to an awesome end user experience where any piece of data could be accessed by either pinky queries or brain queries, but unfortunately we found that federation was not widely supported and was considered experimental and unreliable by experts .
b. click the drop - down list box under the class of application data for which you want to specify an authoritative source (for example: click in the list box under “application response time”) .
time of day—enter the time of the day that you want the data cleanup to happen. the valid format is hh: mm am | pm. for example, 12: 49 am. default: enabled .
make sure it can scale: we have found that data doesn’t grow linearly with the business, but grows superlinearly as technical employees begin building new products and logging new activities on top of the growth of the business .
prefer standard components and methods: there are times when it makes sense to invent a completely new piece of infrastructure, but often times this is not a good use of resources. having intuition about when to build a unique solution and when to adopt an existing solution is important, and that intuition must properly account for the hidden costs of maintenance and support .
enabling data scientists and machine learning engineers beyond the above, looking forward, a key differentiator between checkr and other background check services is our keen focus on automation, something we’ve managed to accomplish in part by building custom machine learning models for tasks such as name matching. reducing burden for development of machine learning models — and as a result, the downstream operational burden — can assist in easing even the most mundane of tasks, such as data extraction .
one of the most serious problems of having a unique infrastructure system was being forced to create custom monitoring and alerting for the cluster. hadoop, hive, and hdfs are complicated systems, prone to bugs across their many knobs and dials. trying to anticipate all failure states and set reasonable pager thresholds proved quite challenging and it also felt that we were solving a solved problem .
source data comes into our system from two main channels: instrumentation in source code that sends events through kafka and production database dumps that are taken using aws point - in - time - restores on rds, then delivered through sqoop .
we’ve talked a lot about registers forming part of a digital ecosystem, providing data good enough to build services on. we’ve also outlined the characteristics of a register and how each characteristic helps registers remain authoritative and up to date. …
further to the remote nature of reads and writes, we had incorrectly chosen to split our data storage across three separate availability zones, within a single region, in aws. furthermore, each availability zone was designated as its own “rack” so the 3 replicas were stored on different racks, therefore remote reads and writes were happening constantly. this was again a design flaw that led to slow data transfers and remote copies happening anytime a machine was lost or a block corrupted .
previously, we described some technical features needed to support the creation of registers. when we talk about the technical details, we often get asked whether a register is like a particular data technology such as a relational database, a graph …
facilitating eng development during engineering development, we frequently need to query our production data to fetch and verify records. nonetheless, querying against our transactional stores has historically been slow, as well as stressful on and potentially destabilizing to our production environment .
presto is extremely fast, not to mention in use at large scale at facebook, airbnb, netflix, and more. given the size of our data (hundreds of gigabytes), we knew presto could handle our use at all levels .
the basic approach (see figure 1) is to show how to value a project by looking at the capabilities to be implemented or updated, the benefits to different parts of the university community, the risks entailed in undertaking the project (or not undertaking it), and the costs (including ongoing maintenance and operational costs) — all based on a fundamental understanding of the different types of infrastructure projects and their purposes .
finally, it is important to know your audience. who are the decision makers who will evaluate your analysis? what measures will resonate with them? if it is faculty, then the most important aspects might be how the infrastructure will aid their ability to do research, obtain research grants, or better teach students. if your audience is the cfo, then an argument with at least some discussion of financial return will be useful .
purges alarms, events, and syslogs according to the settings in the following table. all of these settings are enabled by default. data is deleted on a daily basis. alarm tables are checked hourly, and if the alarm table exceeds the 300, 000 limit ,
solve real problems by listening to your colleagues: empathizing with the data users around the company is an important part of informing our roadmap. in adherence with henry ford’s mantra, we must balance making faster horses vs. building the automobile — but listen to your customers first .
an infrastructure project under review must be compared to other projects. alternatives likely also exist, such as procuring equipment from different vendors or moving to the cloud. the option of doing nothing also has benefits, risks, and costs and should be evaluated with other alternatives. this evaluation should compare all the factors discussed, from general capabilities to benefits to risks and costs. table 3, while not exhaustive, lists many of the criteria that can be used in such a comparison.
a data infrastructure is a digital infrastructure promoting data sharing and consumption . similarly to other infrastructures , it is a structure needed for the operation of a society as well as the services and facilities necessary for an economy to function , the data economy in this case . 
a data infrastructure is a digital infrastructure promoting data sharing and consumption. similarly to other infrastructures, it is a structure needed for the operation of a society as well as the services and facilities necessary for an economy to function, the data economy in this case.
a data infrastructure is a digital infrastructure promoting data sharing and consumption. similarly to other infrastructures, it is a structure needed for the operation of a society as well as the services and facilities necessary for an economy to function, the data economy in this case.