Trending February 2024 # Working And Examples Of Partitionby In Pyspark # Suggested March 2024 # Top 11 Popular

You are reading the article Working And Examples Of Partitionby In Pyspark updated in February 2024 on the website Eastwest.edu.vn. We hope that the information we have shared is helpful to you. If you find the content interesting and meaningful, please share it with your friends and continue to follow and support us for the latest updates. Suggested March 2024 Working And Examples Of Partitionby In Pyspark

Introduction to PySpark partitionBy

PYSPARK partitionBy is a function in PySpark that is used to partition the large chunks of data into smaller units based on certain values. This partitionBy function distributes the data into smaller chunks that are further used for data processing in PySpark. For example, DataFrameWriter Class functions in PySpark that partitions data based on one or multiple column functions.

Start Your Free Software Development Course

Web development, programming languages, Software testing & others

Using partitionBy improves the performance of the data processing and makes the analysis of data easier. The data can be partitioned in memory or disk-based on the requirement we have for data. This article will try to analyze the various method used in PARTITIONBY with the data in PySpark. But, first, let us try to see about PARTITIONBY in some more detail.

The syntax for PYSPARK partitionBy function is:-

b.write.option("header",True).partitionBy("Name").mode("overwrite").csv("path")

b: The data frame used.

write.option: Method to write the data frame with the header being True.

partitionBy: The partitionBy function to be used based on column value needed.

mode: The writing option mode.

csv: The file type and the path where these partition data need to be put in.

Screenshot:

Working of PySpark partitionBy

Let us see somehow PARTITIONBY operation works in PySpark:-

The partitionBy operation works on the data in PySpark by partitioning the data into smaller chunks and saving it either in memory or in the disk in a PySpark data frame. This partition helps in better classification and increases the performance of data in clusters. The partition is based on the column value that decides the number of chunks that need to be partitioned on.

Part files are created that hold the data with the partitioned column name as the folder name in the PySpark. The partitioning allows the data access faster as it will have the data organized way, i.e. in a separate folder and file by which the traversal will be comparatively easier.

All the data are segregated in a common folder with the same data in the same file location needed for columns; this partition can partition the data on single columns as well as multiple columns of a PySpark data frame. Thus, the performance of queries is improved by using the PySpark partition while dealing with huge chunks of data in PySpark.

A success file and a crc file are created to execute the files in the folder given successfully.

Let’s check the creation and working of the partitionBy function with some coding examples.

Example of PySpark partitionBy

Let us see some examples of how partitionBy operation works:-

Let’s start by creating simple data in PySpark.

data1  = [{'Name':'Jhon','ID':21.528,'Add':'USA'},{'Name':'Joe','ID':3.69,'Add':'USA'},{'Name':'Tina','ID':2.48,'Add':'IND'},{'Name':'Jhon','ID':22.22, 'Add':'USA'},{'Name':'Joe','ID':5.33,'Add':'INA'}]

A sample data is created with Name , ID and ADD as the field.

a = sc.parallelize(data1)

RDD is created using sc.parallelize.

b = spark.createDataFrame(a) b.show()

Created Data Frame using Spark.createDataFrame.

Screenshot:

This creates the data frame with the column name as Name, Add, and ID.

Let’s try to partition the data based on Name and store it back in a csv file in a folder.

b.write.option("header",True).partitionBy("Name").mode("overwrite").csv("tmp")

This partitions the data based on Name, and the data is divided into folders. The success and success. crc folder is created with the folder name and the data inside the folder.

The option header true keeps up the header function with it, and the headers are within it. The mode defines the mode under which the data needs to be written. It can be overwritten, append, etc. The column name is written on which the partition needs to be done.

This creates a folder with the name of the folder, and the data is inside that folder.

Screenshot:

Screenshot:

b.write.option("header",True).partitionBy("Name","Add").mode("overwrite").csv("/tmp/")

This will create a folder inside the folder with the name followed by the Add folder.

Screenshot:-

The files are written back in the folder and then can be used for further data analytics purposes.

These are some of the Examples of PARTITIONBY FUNCTION in PySpark.

Note:

partitionBy is a function used to partition the data based on columns in the PySpark data frame.

PySpark partitionBy fastens the queries in a data model.

partitionBy can be used with single as well multiple columns also in PySpark.

partitionBy stores the value in the disk in the form of the part file inside a folder.

partitionBy allows the data movement and shuffling of data around the network.

Conclusion

From the above article, we saw the working of PARTITIONBY in PySpark. Then, from various examples and classifications, we tried to understand how this PARTITIONBY operation happens in PySpark and what are is used at the programming level.

Recommended Articles

This is a guide to PySpark partitionBy. Here we discuss the working of PARTITIONBY in PySpark with various examples and classifications. You may also have a look at the following articles to learn more –

You're reading Working And Examples Of Partitionby In Pyspark

Working And Examples Of Jquery Queue() Function

Introduction to jQuery queue

The jQuery queue() function is used to shows the queue of functions to be executed on the selected element. The jQuery queue() function is a built-in function in jQuery. The queue() function performs on a selector to know how many functions to execute; the queue represents one or more functions waiting to execute. The element has only one queue, the “fx” queue as the default queue in jQuery, but the element can have more than one queue. As the queue() function shows the queued function on the selected element, the dequeue() function removes the function from the queue from the elected element, so both of these functions are sometimes used together.

Start Your Free Software Development Course

Web development, programming languages, Software testing & others

Syntax

$(selector).queue(qName);

qName – This is an optional parameter that specifies the name of the queue. Whereas the default queue is “fx”.

Working of the jQuery queue() function

The jQuery queue() function uses the selector to show the queue of function to be executed on it. The selector can also be an element, id, or class. This function accepts one parameter that is the name of the queue. For example, to a div element, if we apply some functions like animate(), slide(), fade(), and all, so all these functions queued into the queue of the div element, and from the queue one by one all function executed on the div element.

Examples for the jQuery queue() function

Below are the example of the jQuery queue() function to show the length of the queue –

Next, we write the html code to understand the jQuery queue() function more clearly with the following example, where the queue() function is used to show the length of the div queue, as below –

Example #1

Code:

$(document).ready(function() { var div_ele = $(“div”); div_ele.animate( {height : 200, width : 200}, “slow” ); div_ele.animate( {height : 350, width : 350}, “slow” ); div_ele.slideUp(200); div_ele.fadeIn( “fast” ); $( “h3” ).text(“The length of the queue is : ” + div_ele.queue().length ); }); }); jQuery queue() for div element.

Output:

As in the above program, the four functions are applying to the div element, which gets queued into the queue of the div element. Later in the code, the length of this queue displays four, which are the same as the number of the function applied, as we can see in the output.

Example of jQuery queue() function to show the length of function queue which are applied in chaining –

Next, we write the html code to understand the jQuery queue() function, where the queue() function is used to show the length of the queue where functions are applied to div element in chaining, as below –

Example #2

$(document).ready(function() { var div_ele = $(“div”); div_ele.css(“color”, “red”).slideUp(2000).slideDown(2000); alert( “The length of the queue is : ” + div_ele.queue().length ); }); });

Output:

As in the above program, the two functions are applying to the div element in chaining form(slideUp and slideDown), which gets queued into the queue of the div element. Later in the code, the length of this queue displays two, which are the same as the number of the function applied, as we can see in the output.

Example to show the use of queue() and dequeue() function together –

Next, we write the html code to understand the jQuery queue() function, where the queue() function and dequeue() function use together on the div element, as below –

Example #3

$(document).ready(function() { var div_ele = $(“div”); div_ele.css(“color”, “green”).slideUp(2000).slideDown(2000); div_ele.slideUp(2000); $( “h3” ).text( “The length of the queue is : ” + div_ele.queue(“fx”).length ); }); var div_ele = $(“div”); div_ele.dequeue(); $( “h3” ).text( “The length of the queue is : ” + div_ele.queue().length ); }); });

Output:

Conclusion

The jQuery queue function is a built-in function in jQuery, which is used to shows the queue of functions to be executed on the selected elements.

Recommended Articles

This is a guide to the jQuery queue. Here we discuss the Working of the jQuery queue() function and Examples along with the codes and outputs. You may also have a look at the following articles to learn more –

Working And Examples Of Identity Property Column

Introduction to SQL identity

SQL provides us with a property named Identity that is basically a sequence maintained against the column of the table for which that property is used. This property proves to be of immense help when we want to generate the unique identifier values for certain columns. Identity property is mostly used for the columns, which we want to treat as the primary key. As the primary key column must be unique for identification purposes, Identity property helps to assign the auto-generated unique value incrementally for that column.

Start Your Free Data Science Course

Hadoop, Data Science, Statistics & others

The syntax of the IDENTITY property is as follows –

IDENTITY [ ( seed , increment ) ]

where the seed is the initial value of the sequence that will be created against that column for that table, and the increment value will be the step value with which each time the sequence value of IDENTITY will get incremented for each of the newly inserted value.

Sequence and Identity property

SQL sequence is a list of integers. A sequence generates the integers in ascending order and is mostly used to get the unique numbers that are further used for identification purposes. For example, book id in the library table, task id to store the tasks/processes, etc. We can use the Identity property to automatically create a sequence in SQL for a particular column of the table. The only difference between sequence and identity property is that the sequence is user-generated and can be shared between multiple tables, while identity property is system-generated sequence property that is assigned to a single table. This column is most often the column on which the primary key is defined. There are some of the points that you should know about the Identity property/property that are listed below –

There can be only one column in a particular table that can be declared and assigned the Identity property.

The data type of the column to which the Identity property is assigned is mostly integer.

The column which is assigned Identity property needs to be a key/indexed column. This key can be either a primary key or a unique index key.

Working of Identity property column

If declared with IDENTITY(2,1), the value of the Identity column begins with 2 and then it is incremented by 1 whenever a new row is inserted with Identity column value as null or skipping the insertion of this column while inserting a new record in that table. Here, 2 is called the seed, the first parameter, and 1 is the increment value by which the value will increase each time a new row is added.

If we want to add the IDENTITY property to a column that is previously present in the table. Then we can do so by dropping that column of the table and reading it using the IDENTITY property assigned to it with the help of the ALTER TABLE command.

If we try to insert the record by specifying the column’s value, there will be two possible cases. Suppose the value that is inserted in the Identity column does not exist in the table. In that case, SQL will allow its insertion, and the value of the sequence of that Identity column will be set to the value that we inserted. The next value that will be retrieved from that sequence will be equivalent to the current inserted value +increament parameter set the IDENTITY property if IDENTITY_INSERT on that table is ON else, it will throw an error if IDENTITY_INSERT on that table is OFF. In case if the value already exists in the table for the Identity column, an error will be issued saying a column with that value of the Identity column already exists in the table.

If we try to update the value of the Identity column, there will be two cases. Firstly, if the value that we are updating is present in the table, then it will issue the error saying the duplicate-key error for the column, which is declared as Identity by us as it also has an implicit unique index. Secondly, if the value that we are updating for that column does not exist in the table, then that value will be updated for the row, and the sequence value will be set to the current value that we updated if IDENTITY_INSERT on that table is ON else it will throw an error if IDENTITY_INSERT on that table is OFF. Hence, when we will insert the record next time, the value of the sequence that we will get will be our updated value + increment parameter value set while defining the IDENTITY property.

Whenever we delete the row in the last inserted table, then the next value that we will get for the auto-incremented column will not be the same as the value that we deleted. Whether the deleted value will be used or not usually depends upon the table’s storage engine that we have created. For MyISAM and InnoDB engines, the deleted row value of Identity is usually not used, and the new row is entered with the value that is equal to deleted row id + increment parameter value set while defining IDENTITY property. For example, if we deleted the maximum/last inserted row with id column 15, then the next value we will insert will have value 17 instead of 15 if we set the increment value parameter to 2.

Example of SQL identity

Assigning IDENTITY property to new table’s column while creating table –

Let us create one table named educba_ identity1 using the following create table query –

CREATE TABLE educba_identity1 ( id INT NOT NULL IDENTITY(1,1), description VARCHAR(45) DEFAULT NULL );

that gives the following output –

Let us insert some records in the table –

INSERT INTO  educba_identity1(descripion) VALUES("JAVA"),("SQL"),("HIBERNATE"),("MAVEN"),("JAVASCRIPT");

that gives the following output –

SELECT * FROM educba_identity1;

Assigning IDENTITY property to existing table’s column –

CREATE TABLE educba_identity2 ( id INT NOT NULL, description VARCHAR(45) DEFAULT NULL );

that gives the following output –

ALTER TABLE educba_identity2 DROP COLUMN id;

that gives the following output –

ALTER TABLE educba_identity2 ADD id INT IDENTITY(1,1);

Let us add some records in educba_identity2 –

INSERT INTO  educba_identity2(descripion) VALUES("MYSQL"),("HADOOP"),("DATA SCIENCE"),("RUBY"),("NODEJS");

that gives the following output –

SELECT * FROM  educba_identity2;

that gives the following output –

Conclusion – SQL identity

We can use the Identity property to generate the unique values that are incremented by the value specified in the second parameter to the column in the table. Note that one table can contain only one column with the Identity property in SQL.

Recommended Articles

We hope that this EDUCBA information on “SQL identity” was beneficial to you. You can view EDUCBA’s recommended articles for more information.

Working Of Exit Function In Php With Examples

Introduction to PHP exit

Whenever there is a need to terminate the current script along with a message in PHP, we make use of an inbuilt function called exit function in PHP, though exit function is used to terminate the current script, it does not interrupt the object destructors and shut down functions from being executed and exit function takes one parameter namely message where message represents the message that is to be displayed during the termination of the current script by the exit function or this message can also be a status number during the termination of the script by the exit function.

Start Your Free Software Development Course

Syntax:

exit(message)

where,

The message represents the message that is to be displayed during the termination of the current script by the exit function or this message can also be a status number during the termination of the script by the exit function.

Working of exit Function in PHP

Whenever there is a need to terminate the current script along with a message in PHP, we make use of an inbuilt function called exit function in PHP.

Though the exit function is used to terminate the current script, it does not interrupt the object destructors and shut down functions from being executed.

The exit function takes one parameter namely message.

The parameter message represents the message that is to be displayed during the termination of the current script by the exit function or this message can also be a status number during the termination of the script by the exit function.

Examples of PHP exit

Lets us discuss a few examples.

Example #1

PHP program to illustrate the working of exit function where we try to connect to a website through PHP program and if unable to connect, exit the script along with an error message by making use of exit function:

Code:

<?php #a variable called website is used to store the URL of the website that is to be connected. #fopen function is used in read mode to read the contents of the website fopen($website,”r”) #in case if the program is unable to connect to the website and read the contents of the website, the current script is terminated along with an error message using exit function or exit(“Unable to connect to the website: $website “);

Output:

In the above program, a variable called website is used to store the URL of the website that is to be connected. Then fopen function is used to open the website in read mode to read the contents of the website. In case the program fails to connect to the website or fails

Example #2

PHP program to illustrate the working of exit function where we try to connect to a website through PHP program and if unable to connect, exit the script along with an error message by making use of exit function:

Code:

<?php #a variable called website is used to store the URL of the website that is to be connected. #fopen function is used in read mode to read the contents of the website fopen($website,”r”) #in case if the program is unable to connect to the website and read the contents of the website, the current script is terminated along with an error message using exit function or exit(“Unable to connect to the website: $website “);

Output:

In the above program, a variable called website is used to store the URL of the website that is to be connected. Then fopen function is used to open the website in read mode to read the contents of the website. In case the program fails to connect to the website or fails to read the contents of the website, then an error message is displayed while terminating the script by making use of the exit function which is displayed as the output on the screen.

Example #3

PHP program to illustrate the working of exit function where we try to open a file by specifying the path to the location of the file through PHP program and if unable to open the file, exit the script along with an error message by making use of exit function:

Code:

<?php #a variable called filepath is used to store the path to the location of the file that is to be opened. $filepath = “C:/Users/admin/Desktop/check”; #fopen function is used in read mode to read the contents of the file present at the location specified fopen($filepath, “r”) #in case if the program is unable to open the file present at the location specified and read the contents of the file, the current script is terminated along with an error message using exit function or exit(“Unable to open the file present at the location $filepath”);

Output:

In the above program, a variable called filepath is used to store the path to the location of the file to be opened. Then fopen function is used to open the file in read mode to read the contents of the file from the path that specifies the location of the file. In case the program fails to open the file present at the location specified by the path of the file, the program terminates displaying an error message by making use of the exit function which is displayed as the output on the screen. The output is shown in the snapshot above.

Conclusion

In this article, we have learnt the concept of exit function in PHP through definition, syntax, working of exit function through programming examples, and their outputs.

Recommended Articles

This is a guide to PHP exit. Here we discuss the introduction, syntax, and working of the exit function in PHP along with examples and code implementation. You may also have a look at the following articles to learn more –

Working Of Scala Filter With The Programming Examples

Introduction to Scala filter

Scala filter is a method that is used to select the values in an elements or collection by filtering it with a certain condition. The Scala filter method takes up the condition as the parameter which is a Boolean value and returns the result after filtering over that condition. Whatever values that satisfies that condition is given as the output result and the one not is eradicated from the list. Scala filter method filters the data on a Boolean condition and produces the result.

Start Your Free Software Development Course

Web development, programming languages, Software testing & others

Syntax:

It returns a new list with the filter condition containing the values that satisfies the condition.

Working of Scala filter with Examples

Scala filter checks for the condition whatever is mentioned and filters the data then accordingly. It first checks the condition for the Boolean value if it is true it filters the data based on that particular value and then publishes the result. The filter function walks through the all elements in a collection, we just need to pass the logic or the algorithm by which we need to use the filter method. We can pass complex logics as well as methods also inside an filter method. We can also use filter in the same code or logic as many times as we want to.

Example #1

Code:

a: List[Int] = List(3, 4, 5, 6, 7, 8) Here we created an List with the values stored in a and applied the filter method over that, that gives up all the values greater than 6. res4: List[Int] = List(7, 8) Example #2

Code:

b: List[String] = List(Arpit, Anand) res11: List[String] = List(Arpit) res12: List[String] = List()

So here we saw if the particular condition is not fulfilled then an empty list value is returned.

Output:

Example #3

Using more than more filter over the same collection.

Code:

a: List[Int] = List(3, 4, 5, 6, 7, 8) res14: List[Int] = List(7) res17: List[Int] = List()

If any of the condition fails it returns an null or empty value.

Output:

Example #4

We can even pass the map function also over the filtered values to get desired results.

Code:

a: List[Int] = List(3, 4, 5, 6, 7, 8) res22: List[Int] = List(14, 16)

This first filters out the data and then a map operation is performed with the filtered data.

Example #5

Using Filter Operation with Map Function.

Code:

A Map is created with two values in it. res31: scala.collection.immutable.Map[String,Int] = Map() We use the filter condition to check and if it satisfies it produces the result or else Null is returned. res33: scala.collection.immutable.Map[String,Int] = Map()

Output:

So from these above examples we saw how we can use the filter over the map Function. We can also use it with other collections in Scala.

Example #6

Some More Operations with Filter Function.

a. We can use the various methods with the filter function in it, it just works like it takes up the filtered value and produces the result.

Using Zip with Filter:

Code:

a: List[Int] = List(3, 4, 5, 6, 7, 8) b: List[Int] = List(6, 7, 89) res36: List[(Int, Int)] = List((7,6), (8,7)) res37: List[(Int, Int)] = List((5,6), (6,7), (7,89))

b. We can also check the length after applying the filter function and traverse the new collection with the values in it.

Code:

res38: Int = 2 res39: Int = 4 5 6 7 8

c. We can also find the first occurrence of any condition inside the filter method with the help of find function with filter.

Code:

a: List[Int] = List(3, 4, 5, 6, 7, 8) res49: Option[Int] = Some(5) res50: Option[Int] = Some(7) res51: Option[Int] = Some(6)

d. So just as we do operation over list or map or any collection class in Scala we same can do by filtering the value thereafter using the .filter method in Scala.

Methods like Max, Min, Sum, etc can be used in the same way we use it with the other collection class in Scala.

Code:

res52: Int = 8 res53: Int = 4 res54: Int = 30

Output:

So from the above article we found how filter operation can be done over the collection class and various methods can be applied over that to achieve the result.

Conclusion

From the above article we saw how we can use the Scala filter method and checked the syntax and return type for the same. We also saw with the help of examples how the filter operation is applied to a collection and the methods we can use over the filter operation. So by this we can conclude that Scala filter method is an important method in Scala object oriented programming to filter out the data in a code efficiently.

Recommended Articles

We hope that this EDUCBA information on “Scala filter” was beneficial to you. You can view EDUCBA’s recommended articles for more information.

Usage And Examples Of Db2 Merge In Detail

Introduction to DB2 merge

DB2 MERGE statement is used to perform multiple operations on a particular table that is considered as the main or target table. The operations that are needed to be performed on that table depend on whether the records of that table are matching with the reference table also called as the source table.

Start Your Free Data Science Course

The match is made on the basis of certain conditions and values that are maintained in both tables. These conditions can be either a single condition or even multiple conditions that are separated by using the logical operators like AND and OR operators in the query statement. Further, the MERGE statement also allows performing certain operations when records are matched and some other operations when records aren’t matched.

In this article, we will study the syntax of the MERGE statement in DB2 and its implementation with the help of multiple examples demonstrating the usage of the MERGE statement.

Syntax 

The syntax of the MERGE statement used in DB2 is as shown below –

,….]

In the above syntax, we have the MERGE INTO clause used for merging the main or target table with respect to changes that are being made to the source or reference table. We can also use views in place of tables for target and source ones. Further, we can have as many WHEN MATCHED and NOT MATCHED conditions as required for performing different operations as shown above in the syntax by adding ….

Usage

We can make the use of the MERGE statement in DB2 in any of the application programs by embedding it or it can be invocated with the help of dynamic SQL statements which are nothing but the statements that can be executed and are prepared dynamically on the run time depending on the requirement. We can make use of MERGE statements with tables as well as views. Whenever we try to do the operations on the views in the MERGE statement, the corresponding reflection of the operations can be seen in the tables associated with the views ass well.

Example

Let us consider one example where we will be using the MERGE statement to update the table named bank_account_balance which is the target table and the main table which stores the data related to all the accounts and the balance of the corresponding account ids. This table needs to be updated whenever the table sum_transaction_amount which stores the data related to all the transactions that are made on all the accounts whenever any transaction is performed.

VALUES (at.account_id, at.sum_transaction_amount);

The execution of the above MERGE statement gives the following output

VALUES (do.item_no, do.details, do.number_of_items)

VALUES (l.worker_id, l.mobile_no, l.workplace)

The execution of the above MERGE statement gives the following output –

Conclusion – DB2 merge

We can make use of the MERGE statement in DB2 to perform different transactions on the target table based on the operations that are being performed on the source table in order to synchronize the source and target tables with each other.

Recommended Articles

This is a guide to DB2 merge. Here we discuss the syntax of the MERGE statement in DB2 and its implementation with the help of multiple examples demonstrating the usage of the MERGE statement. You may also have a look at the following articles to learn more –

Update the detailed information about Working And Examples Of Partitionby In Pyspark on the Eastwest.edu.vn website. We hope the article's content will meet your needs, and we will regularly update the information to provide you with the fastest and most accurate information. Have a great day!