在上一篇文章中，介绍了如何使用python导入数据，导入数据后的第二步往往就是数据清洗，下面我们来看看如何使用pandas进行数据清洗工作

导入相关库

import pandas as pd

dataframe = pd.read_csv(r'C:/Users/DELL/data-science-learning/python数据分析笔记/探索性数据分析/train.csv')
dataframe.head(5)
PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked
0 1 0 3 Braund, Mr. Owen Harris male 22.0 1 0 A/5 21171 7.2500 NaN S
1 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Th… female 38.0 1 0 PC 17599 71.2833 C85 C
2 3 1 3 Heikkinen, Miss. Laina female 26.0 0 0 STON/O2. 3101282 7.9250 NaN S
3 4 1 1 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1 0 113803 53.1000 C123 S
4 5 0 3 Allen, Mr. William Henry male 35.0 0 0 373450 8.0500 NaN S

🥇1.总览数据

  • 查看数据维度
dataframe.shape


(891, 12)
  • 描述性统计分析
dataframe.describe()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

PassengerId Survived Pclass Age SibSp Parch Fare
count 891.000000 891.000000 891.000000 714.000000 891.000000 891.000000 891.000000
mean 446.000000 0.383838 2.308642 29.699118 0.523008 0.381594 32.204208
std 257.353842 0.486592 0.836071 14.526497 1.102743 0.806057 49.693429
min 1.000000 0.000000 1.000000 0.420000 0.000000 0.000000 0.000000
25% 223.500000 0.000000 2.000000 20.125000 0.000000 0.000000 7.910400
50% 446.000000 0.000000 3.000000 28.000000 0.000000 0.000000 14.454200
75% 668.500000 1.000000 3.000000 38.000000 1.000000 0.000000 31.000000
max 891.000000 1.000000 3.000000 80.000000 8.000000 6.000000 512.329200

🥈2.筛选数据

  • 过滤所有女性和年龄大于60岁的乘客
dataframe[(dataframe['Sex'] == 'female') & (dataframe['Age']>=60)]

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked
275 276 1 1 Andrews, Miss. Kornelia Theodosia female 63.0 1 0 13502 77.9583 D7 S
366 367 1 1 Warren, Mrs. Frank Manley (Anna Sophia Atkinson) female 60.0 1 0 110813 75.2500 D37 C
483 484 1 3 Turkula, Mrs. (Hedwig) female 63.0 0 0 4134 9.5875 NaN S
829 830 1 1 Stone, Mrs. George Nelson (Martha Evelyn) female 62.0 0 0 113572 80.0000 B28 NaN

可以看出，一共有四名年龄大于60岁的女性乘客

🥉3.替换数据

  • 将female换成woman，将male换成man
dataframe['Sex'].replace(['female','male'],['woman','man']).head(5)


0      man
1    woman
2    woman
3    woman
4      man
Name: Sex, dtype: object

🏅4.更改列名

  • 查看所有列名
dataframe.columns


Index(['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp',       'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked'],
      dtype='object')
  • 重命名列
PassengerId Survived Passenger Class Name Sex Age SibSp Parch Ticket Fare Cabin Embarked
0 1 0 3 Braund, Mr. Owen Harris male 22.0 1 0 A/5 21171 7.2500 NaN S
1 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Th… female 38.0 1 0 PC 17599 71.2833 C85 C
2 3 1 3 Heikkinen, Miss. Laina female 26.0 0 0 STON/O2. 3101282 7.9250 NaN S
3 4 1 1 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1 0 113803 53.1000 C123 S
4 5 0 3 Allen, Mr. William Henry male 35.0 0 0 373450 8.0500 NaN S 
dataframe.rename(columns={'Pclass':'Passenger Class','Sex':'Gender'}).head()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

PassengerId Survived Passenger Class Name Gender Age SibSp Parch Ticket Fare Cabin Embarked
0 1 0 3 Braund, Mr. Owen Harris male 22.0 1 0 A/5 21171 7.2500 NaN S
1 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Th… female 38.0 1 0 PC 17599 71.2833 C85 C
2 3 1 3 Heikkinen, Miss. Laina female 26.0 0 0 STON/O2. 3101282 7.9250 NaN S
3 4 1 1 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1 0 113803 53.1000 C123 S
4 5 0 3 Allen, Mr. William Henry male 35.0 0 0 373450 8.0500 NaN S

🥇5.查找唯一值

在pandas中，我们可以使用unique()查找唯一值

# 查找唯一值
dataframe['Sex'].unique()


array(['male', 'female'], dtype=object)


# 显示唯一值出现的个数
dataframe['Sex'].value_counts()


male      577
female    314
Name: Sex, dtype: int64


# 查找类型票的数量
dataframe['Pclass'].value_counts()


3    491
1    216
2    184
Name: Pclass, dtype: int64


# 查找唯一值的种类
dataframe['Pclass'].nunique()



3

🥈6.查找缺失值

# 查找空数据
dataframe[dataframe['Age'].isnull()].head()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked
5 6 0 3 Moran, Mr. James male NaN 0 0 330877 8.4583 NaN Q
17 18 1 2 Williams, Mr. Charles Eugene male NaN 0 0 244373 13.0000 NaN S
19 20 1 3 Masselmani, Mrs. Fatima female NaN 0 0 2649 7.2250 NaN C
26 27 0 3 Emir, Mr. Farred Chehab male NaN 0 0 2631 7.2250 NaN C
28 29 1 3 O’Dwyer, Miss. Ellen “Nellie” female NaN 0 0 330959 7.8792 NaN Q

pandas没有NaN 如果想要处理的话必须导入numpy包

import numpy as np
dataframe['Sex'].replace('male',np.nan).head()


0       NaN
1    female
2    female
3    female
4       NaN
Name: Sex, dtype: object

🥉7.删除列或行

# 删除一列，采用drop方法，并传入参数axis
dataframe.drop('Age',axis=1).head()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

PassengerId Survived Pclass Name Sex SibSp Parch Ticket Fare Cabin Embarked
0 1 0 3 Braund, Mr. Owen Harris male 1 0 A/5 21171 7.2500 NaN S
1 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Th… female 1 0 PC 17599 71.2833 C85 C
2 3 1 3 Heikkinen, Miss. Laina female 0 0 STON/O2. 3101282 7.9250 NaN S
3 4 1 1 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 1 0 113803 53.1000 C123 S
4 5 0 3 Allen, Mr. William Henry male 0 0 373450 8.0500 NaN S 
#删除行
dataframe.drop(1)


# 删除重复行 使用subset参数指明要删除的列
dataframe.drop_duplicates(subset='Sex').head()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

Name PClass Age Sex Survived SexCode
0 Allen, Miss Elisabeth Walton 1st 29.0 female 1 1
2 Allison, Mr Hudson Joshua Creighton 1st 30.0 male 0 0

🏅8. groupby分组

  • 计算男性和女性的平均值

==思路一==，将所有男性和女性的条件进行选取分别计算

man = dataframe[dataframe['Sex']=='male']
woman = dataframe[dataframe['Sex']=='female']
print(man.mean())
print(woman.mean())


Age         31.014338
Survived     0.166863
SexCode      0.000000
dtype: float64
Age         29.396424
Survived     0.666667
SexCode      1.000000
dtype: float64

==思路二==，用groupby方法简化


dataframe.groupby('Sex').mean()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

Age Survived SexCode
Sex
— — — —
female 29.396424 0.666667 1.0
male 31.014338 0.166863 0.0 
# 按行分组，计算行数
dataframe.groupby('Sex')['Name'].count()


Sex
female    462
male      851
Name: Name, dtype: int64


dataframe.groupby(['Sex','Survived']).mean()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

PassengerId Pclass Age SibSp Parch Fare
Sex Survived
— — — — — — — —
female 0 434.851852 2.851852 25.046875 1.209877 1.037037 23.024385
1 429.699571 1.918455 28.847716 0.515021 0.515021 51.938573
male 0 449.121795 2.476496 31.618056 0.440171 0.207265 21.960993
1 475.724771 2.018349 27.276022 0.385321 0.357798 40.821484

🥇9.按照时间段来进行分组

  • 使用resample参数来进行取样本
# 创建时期范围
time_index = pd.date_range('06/06/2017', periods=100000, freq='30S') # periods表示有多少数据，freq表示步长


dataframe = pd.DataFrame(index=time_index)


# 创建一个随机变量
dataframe['Sale_Amout'] = np.random.randint(1, 10, 100000)


# resample 参数，按周对行分组，计算每一周的总和
dataframe.resample('W').sum()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

Sale_Amout
2017-06-11 86292
2017-06-18 100359
2017-06-25 100907
2017-07-02 100868
2017-07-09 100522
2017-07-16 10478 
# 使用resample可以按一组时间间隔来进行分组，然后计算每一个时间组的某个统计量
dataframe.resample('2W').mean()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

Sale_Amout
2017-06-11 4.993750
2017-06-25 4.991716
2017-07-09 4.994792
2017-07-23 5.037500 
dataframe.resample('M').count()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

Sale_Amout
2017-06-30 72000
2017-07-31 28000 
# resample默认是以最后一个数据作 使用label参数可以进行调整
dataframe.resample('M', label='left').count()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

Sale_Amout
2017-05-31 72000
2017-06-30 28000

🥈10.遍历一个列的数据

dataframe = pd.read_csv(url)


# 以大写的形势打印前两行的名字
for name in dataframe['Name'][0:2]:
    print(name.upper())


ALLEN, MISS ELISABETH WALTON
ALLISON, MISS HELEN LORAINE

🥉11.对一列的所有元素应用某个函数

def uppercase(x):
    return x.upper()


dataframe['Name'].apply(uppercase)[0:2]


0    ALLEN, MISS ELISABETH WALTON
1     ALLISON, MISS HELEN LORAINE
Name: Name, dtype: object

🏅12. pandas高级函数

dataframe.groupby('Sex').apply(lambda x:x.count())

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

Name PClass Age Sex Survived SexCode
Sex
— — — — — — —
female 462 462 288 462 462 462
male 851 851 468 851 851 851

通过联合使用groupby 和apply，我们就能计算自定义的统计量 例如上面我们发现age、cabin具有大量的缺失值

🥇13. 连接多个Dataframe

data_a = {'id':['1', '2', '3'],
          'first': ['Alex', 'Amy', 'Allen'],
          'last': ['Anderson', 'Ackerman', 'Ali']}
dataframe_a = pd.DataFrame(data_a, columns=['id','first', 'last'])


data_b = {'id':['4', '5', '6'],
          'first': ['Billy', 'Brian', 'Bran'],
          'last': ['Bonder', 'Black', 'Balwner']}
dataframe_b = pd.DataFrame(data_b, columns=['id','first', 'last'])


pd.concat([dataframe_a, dataframe_b], axis=0)#在行的方向进行

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

id first last
0 1 Alex Anderson
1 2 Amy Ackerman
2 3 Allen Ali
0 4 Billy Bonder
1 5 Brian Black
2 6 Bran Balwner 
pd.concat([dataframe_a, dataframe_b], axis=1)#在列的方向进行

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

id first last id first last
0 1 Alex Anderson 4 Billy Bonder
1 2 Amy Ackerman 5 Brian Black
2 3 Allen Ali 6 Bran Balwner 
# 也可以用append方法进行添加
c = pd.Series([10, 'Chris', 'Chillon'], index=['id','first','last'])


dataframe.append(c, ignore_index=True)#如果c原来有名字忽略

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

id first last
0 1 Alex Anderson
1 2 Amy Ackerman
2 3 Allen Ali
3 10 Chris Chillon

如果你对Python感兴趣，想要学习python，这里给大家分享一份Python全套学习资料，都是我自己学习时整理的，希望可以帮到你，一起加油！

😝有需要的小伙伴，可以点击下方链接免费领取或者V扫描下方二维码免费领取🆓
Python全套学习资料

在这里插入图片描述

1️⃣零基础入门

① 学习路线

对于从来没有接触过Python的同学，我们帮你准备了详细的学习成长路线图。可以说是最科学最系统的学习路线，你可以按照上面的知识点去找对应的学习资源，保证自己学得较为全面。
在这里插入图片描述

② 路线对应学习视频

还有很多适合0基础入门的学习视频，有了这些视频，轻轻松松上手Python~
在这里插入图片描述

③练习题

每节视频课后，都有对应的练习题哦，可以检验学习成果哈哈！
在这里插入图片描述

2️⃣国内外Python书籍、文档

① 文档和书籍资料

在这里插入图片描述

3️⃣Python工具包+项目源码合集

①Python工具包

学习Python常用的开发软件都在这里了！每个都有详细的安装教程，保证你可以安装成功哦！
在这里插入图片描述

②Python实战案例

光学理论是没用的，要学会跟着一起敲代码，动手实操，才能将自己的所学运用到实际当中去，这时候可以搞点实战案例来学习。100+实战案例源码等你来拿！
在这里插入图片描述

③Python小游戏源码

如果觉得上面的实战案例有点枯燥，可以试试自己用Python编写小游戏，让你的学习过程中增添一点趣味！
在这里插入图片描述

4️⃣Python面试题

我们学会了Python之后，有了技能就可以出去找工作啦！下面这些面试题是都来自阿里、腾讯、字节等一线互联网大厂，并且有阿里大佬给出了权威的解答，刷完这一套面试资料相信大家都能找到满意的工作。
在这里插入图片描述
在这里插入图片描述

5️⃣Python兼职渠道

而且学会Python以后，还可以在各大兼职平台接单赚钱，各种兼职渠道+兼职注意事项+如何和客户沟通，我都整理成文档了。
在这里插入图片描述

上述所有资料 ⚡️ ，朋友们如果有需要的，可以扫描下方👇👇👇二维码免费领取🆓
在这里插入图片描述

# python # 数据分析 # 开发语言 # 爬虫 # 学习 # 设计模式 # 数据挖掘
Logo
DAMO开发者矩阵

DAMO开发者矩阵，由阿里巴巴达摩院和中国互联网协会联合发起，致力于探讨最前沿的技术趋势与应用成果，搭建高质量的交流与分享平台，推动技术创新与产业应用链接，围绕“人工智能与新型计算”构建开放共享的开发者生态。

更多推荐

cover

KingbaseES数据库：医疗信创潮涌沅江，国产化信创打造医疗新质生产力

avatar DAMO开发者矩阵
cover

告别迁移焦虑：金仓KDMS V4带你轻松搞定数据库国产化替代

avatar DAMO开发者矩阵
cover

KingbaseES数据库在常德二院全栈国产化信创中产生新质生产力,医疗信创的部署如何实现，如何操作？

avatar DAMO开发者矩阵