基于hadoop hive大数据的电影数据分析系统实战，大数据分析系统，电影数据分析系统

基于大数据的电影数据分析系统实战

一、项目概述

本项目是一个完整的电影数据分析系统，整合了大数据处理、后端API服务和前端可视化展示三大模块。系统通过对电影数据的采集、清洗、分析和可视化，为电影行业从业者提供数据支持。

1.1 项目架构

项目采用分层架构设计，主要包含三个核心模块：

movie-analysis/
├── data/ # 数据处理模块
│ ├── create_table.sql # MySQL表结构
│ ├── spark_movie_analysis.py # Spark数据分析脚本
│ ├── create_and_load_hive_table.hql # Hive表创建
│ └── import_to_mysql.sh # Sqoop导入脚本
├── movie-analysis-api/ # 后端API模块
│ ├── src/main/java/
│ │ └── com/example/movieanalysis/
│ │ ├── controller/ # REST控制器
│ │ ├── service/ # 业务逻辑层
│ │ ├── mapper/ # 数据访问层
│ │ └── entity/ # 实体类
│ └── pom.xml # Maven配置
└── movie-analysis-dashboard/ # 前端可视化模块
├── src/
│ ├── components/ # 图表组件
│ ├── views/ # 页面视图
│ ├── utils/ # 工具函数
│ └── router/ # 路由配置
└── package.json # 依赖配置

1.2 技术栈

二、数据层设计与实现

2.1 数据模型设计

系统设计的电影数据模型包含19个字段，涵盖了电影的基本信息、财务数据、评价数据等多个维度：

CREATE TABLE IF NOT EXISTS movies (
id INT AUTO_INCREMENT PRIMARY KEY,
Title VARCHAR(255), — 电影标题
Year INT, — 发行年份
Director VARCHAR(255), — 导演
Duration INT, — 时长(分钟)
Rating DECIMAL(3,1), — 评分
Votes INT, — 投票数
Description TEXT, — 描述
Language VARCHAR(100), — 语言
Country VARCHAR(100), — 制片国家
Budget_USD DECIMAL(15,2), — 预算(美元)
BoxOffice_USD DECIMAL(15,2), — 票房(美元)
Genre VARCHAR(50), — 类型
Production_Company VARCHAR(255), — 制作公司
Content_Rating VARCHAR(20), — 内容分级
Lead_Actor VARCHAR(255), — 主演
Num_Awards INT, — 获奖数
Critic_Reviews INT — 评论数
);

为了优化查询性能，系统为常用查询字段创建了索引：

CREATE INDEX idx_genre ON movies(Genre);
CREATE INDEX idx_year ON movies(Year);
CREATE INDEX idx_rating ON movies(Rating);

2.2 Hive数据仓库

Hive作为数据仓库层，提供了数据存储和查询能力。通过HQL脚本创建表结构并加载数据：

— 创建Hive表
CREATE TABLE IF NOT EXISTS movies (
movie_id INT,
title STRING,
release_year INT,
genre STRING,
director STRING,
rating FLOAT,
runtime INT,
revenue DECIMAL(18,2),
budget DECIMAL(18,2),
language STRING,
country STRING
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS TEXTFILE;

Hive支持多种数据加载方式：

— 从本地文件系统加载
LOAD DATA LOCAL INPATH '/path/to/movies.csv' OVERWRITE INTO TABLE movies;

— 从HDFS加载
LOAD DATA INPATH '/user/hadoop/movies.csv' OVERWRITE INTO TABLE movies;

2.3 Spark数据分析

Spark用于大规模数据的批处理和分析。系统实现了五大核心分析功能：

2.3.1 数据清洗

cleaned_df = movies_df.na.fill({
"Rating": 0.0,
"Votes": 0,
"Budget_USD": 0,
"BoxOffice_USD": 0,
"Num_Awards": 0,
"Critic_Reviews": 0
})

2.3.2 按年份统计电影数量

movies_by_year = cleaned_df.groupBy("Year") \\
.agg(count("*").alias("MovieCount")) \\
.orderBy("Year")

2.3.3 按类型统计电影数量

genre_counts = movies_with_genres \\
.select(explode(col("GenreArray")).alias("Genre")) \\
.groupBy(trim(col("Genre")).alias("Genre")) \\
.agg(count("*").alias("MovieCount")) \\
.orderBy(desc("MovieCount"))

2.3.4 计算投资回报率(ROI)

roi_df = cleaned_df \\
.filter((col("Budget_USD") > 0) & (col("BoxOffice_USD") > 0)) \\
.withColumn(
"ROI",
(col("BoxOffice_USD") – col("Budget_USD")) / col("Budget_USD")
) \\
.select("Title", "Year", "Budget_USD", "BoxOffice_USD", "ROI") \\
.orderBy(desc("ROI"))

2.3.5 结果持久化

# 创建年份统计表
spark.sql("DROP TABLE IF EXISTS bigdata.movies_by_year")
movies_by_year.write.saveAsTable("bigdata.movies_by_year")

# 创建类型统计表
spark.sql("DROP TABLE IF EXISTS bigdata.movies_by_genre")
genre_counts.write.saveAsTable("bigdata.movies_by_genre")

2.4 Sqoop数据迁移

Sqoop用于在Hadoop和关系型数据库之间传输数据。系统提供了从Hive导出数据到MySQL的脚本：

sqoop export \\
–connect jdbc:mysql://localhost:3306/bigdata \\
–username root \\
–password root \\
–table movies \\
–export-dir /user/hive/warehouse/bigdata.db/movies \\
–input-fields-terminated-by '\\001' \\
–input-lines-terminated-by '\\n' \\
–input-null-string '\\\\N' \\
–input-null-non-string '\\\\N' \\
–num-mappers 4

三、后端API设计与实现

3.1 项目结构

后端采用标准的Spring Boot分层架构：

com.example.movieanalysis/
├── aop/ # 切面编程
│ └── LoggingAspect.java # 日志切面
├── common/ # 通用类
│ ├── ApiResponse.java # API响应封装
│ └── ResponseResult.java # 响应结果
├── config/ # 配置类
│ └── WebMvcConfig.java # Web配置
├── controller/ # 控制器
│ ├── MovieAnalysisController.java # 分析控制器
│ └── MovieController.java # 电影控制器
├── entity/ # 实体类
│ └── Movie.java # 电影实体
├── mapper/ # 数据访问层
│ └── MovieMapper.java # 电影Mapper
└── service/ # 服务层
├── MovieAnalysisService.java # 分析服务接口
└── impl/
└── MovieAnalysisServiceImpl.java # 分析服务实现

3.2 实体类设计

使用Lombok简化实体类代码：

@Data
@TableName("movies")
public class Movie {
private Integer id;
private String title;
private Integer year;
private String director;
private Integer duration;
private BigDecimal rating;
private Integer votes;
private String description;
private String language;
private String country;
private BigDecimal budgetUsd;
@TableField("BoxOffice_USD")
private BigDecimal boxOfficeUsd;
private String genre;
private String productionCompany;
private String contentRating;
private String leadActor;
private Integer numAwards;
private Integer criticReviews;
}

3.3 RESTful API设计

系统提供了12个分析接口，覆盖了电影数据的多个维度：

@RestController
@RequestMapping("/api/analysis")
public class MovieAnalysisController {

@GetMapping("/by-genre")
public ApiResponse<Map<String, Long>> analyzeByGenre() {
return ApiResponse.success(movieAnalysisService.analyzeMoviesByGenre());
}

@GetMapping("/by-year")
public ApiResponse<Map<Integer, Long>> analyzeByYear() {
return ApiResponse.success(movieAnalysisService.analyzeMoviesByYear());
}

@GetMapping("/by-rating")
public ApiResponse<Map<String, Long>> analyzeByRating() {
return ApiResponse.success(movieAnalysisService.analyzeMoviesByRating());
}

@GetMapping("/by-country")
public ApiResponse<Map<String, Long>> analyzeByCountry() {
return ApiResponse.success(movieAnalysisService.analyzeByCountry());
}

@GetMapping("/by-language")
public ApiResponse<Map<String, Long>> analyzeByLanguage() {
return ApiResponse.success(movieAnalysisService.analyzeMoviesByLanguage());
}

@GetMapping("/by-duration")
public ApiResponse<Map<String, Long>> analyzeByDuration() {
return ApiResponse.success(movieAnalysisService.analyzeMoviesByDuration());
}

@GetMapping("/budget-boxoffice")
public ApiResponse<List<Map<String, Object>>> analyzeBudgetAndBoxOffice() {
return ApiResponse.success(movieAnalysisService.analyzeMoviesByBudgetAndBoxOffice());
}

@GetMapping("/by-director")
public ApiResponse<Map<String, Long>> analyzeByDirector() {
return ApiResponse.success(movieAnalysisService.analyzeByDirector());
}

@GetMapping("/by-content-rating")
public ApiResponse<Map<String, Long>> analyzeByContentRating() {
return ApiResponse.success(movieAnalysisService.analyzeByContentRating());
}

@GetMapping("/by-awards")
public ApiResponse<Map<String, Long>> analyzeByAwards() {
return ApiResponse.success(movieAnalysisService.analyzeByAwards());
}

@GetMapping("/by-critic-reviews")
public ApiResponse<Map<String, Long>> analyzeByCriticReviews() {
return ApiResponse.success(movieAnalysisService.analyzeByCriticReviews());
}
}

3.4 业务逻辑实现

使用Java 8 Stream API进行数据处理，代码简洁高效：

3.4.1 按类型分析

@Override
public Map<String, Long> analyzeMoviesByGenre() {
List<Movie> movies = movieMapper.selectList(null);
return movies.stream()
.filter(movie -> movie.getGenre() != null)
.collect(Collectors.groupingBy(
Movie::getGenre,
Collectors.counting()
));
}

3.4.2 按评分区间分析

@Override
public Map<String, Long> analyzeMoviesByRating() {
List<Movie> movies = movieMapper.selectList(null);
return movies.stream()
.filter(movie -> movie.getRating() != null)
.collect(Collectors.groupingBy(
movie -> {
BigDecimal rating = movie.getRating();
if (rating.compareTo(new BigDecimal("9")) >= 0) return "9+";
else if (rating.compareTo(new BigDecimal("8")) >= 0) return "8-9";
else if (rating.compareTo(new BigDecimal("7")) >= 0) return "7-8";
else if (rating.compareTo(new BigDecimal("6")) >= 0) return "6-7";
else return "<6";
},
Collectors.counting()
));
}

3.4.3 按导演分析（Top 20）

@Override
public Map<String, Long> analyzeMoviesByDirector() {
List<Movie> movies = movieMapper.selectList(null);
return movies.stream()
.filter(movie -> movie.getDirector() != null)
.collect(Collectors.groupingBy(
Movie::getDirector,
Collectors.counting()
))
.entrySet().stream()
.sorted(Map.Entry.<String, Long>comparingByValue().reversed())
.limit(20)
.collect(Collectors.toMap(
Map.Entry::getKey,
Map.Entry::getValue,
(oldValue, newValue) -> oldValue,
LinkedHashMap::new
));
}

3.4.4 预算与票房分析

@Override
public List<Map<String, Object>> analyzeMoviesByBudgetAndBoxOffice() {
List<Movie> movies = movieMapper.selectList(null);
return movies.stream()
.filter(movie -> movie.getBudgetUsd() != null && movie.getBoxOfficeUsd() != null)
.map(movie -> {
Map<String, Object> result = new HashMap<>();
result.put("budget", movie.getBudgetUsd().doubleValue());
result.put("boxOffice", movie.getBoxOfficeUsd().doubleValue());
result.put("title", movie.getTitle());
return result;
})
.collect(Collectors.toList());
}

3.5 配置文件

server.port=8080

spring.datasource.url=jdbc:mysql://192.168.199.101:3306/bigdata?useSSL=false&serverTimezone=UTC
spring.datasource.username=root
spring.datasource.password=123456
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver

mybatis-plus.mapper-locations=classpath*:mapper/**/*.xml
mybatis-plus.type-aliases-package=com.example.movieanalysis.entity
mybatis-plus.configuration.map-underscore-to-camel-case=true

四、前端可视化设计与实现

4.1 项目结构

前端采用Vue 3组合式API开发：

movie-analysis-dashboard/
├── src/
│ ├── assets/ # 静态资源
│ │ └── css/
│ │ └── main.css # 全局样式
│ ├── components/ # 图表组件
│ │ ├── ChartAwards.vue # 奖项图表
│ │ ├── ChartContentRating.vue # 内容分级图表
│ │ ├── ChartCountry.vue # 国家分布图表
│ │ ├── ChartCriticReviews.vue # 评论图表
│ │ ├── ChartDirector.vue # 导演图表
│ │ ├── ChartDuration.vue # 时长图表
│ │ ├── ChartGenre.vue # 类型图表
│ │ ├── ChartLanguage.vue # 语言图表
│ │ ├── ChartRating.vue # 评分图表
│ │ ├── ChartRevenue.vue # 票房图表
│ │ ├── ChartWordCloud.vue # 词云图表
│ │ └── ChartYear.vue # 年份图表
│ ├── composables/ # 组合式函数
│ │ └── useChartBase.js # 图表基础逻辑
│ ├── router/ # 路由配置
│ │ └── index.js
│ ├── store/ # 状态管理
│ │ └── index.js
│ ├── utils/ # 工具函数
│ │ ├── api.js # API封装
│ │ └── echarts.js # ECharts配置
│ ├── views/ # 页面视图
│ │ ├── AboutView.vue
│ │ ├── Dashboard.vue
│ │ ├── DashboardView.vue
│ │ └── HomeView.vue
│ ├── App.vue # 根组件
│ └── main.js # 入口文件
├── index.html # HTML模板
├── package.json # 依赖配置
└── vite.config.js # Vite配置

4.2 API封装

使用Axios封装HTTP请求，统一处理响应：

import axios from 'axios'

const apiClient = axios.create({
baseURL: 'http://localhost:8080',
withCredentials: false,
headers: {
Accept: 'application/json',
'Content-Type': 'application/json'
}
})

apiClient.interceptors.response.use(
response => response,
error => {
return Promise.reject(error)
}
)

export default {
getGenres() {
return apiClient.get('/api/analysis/by-genre')
.then(response => {
return response.data.data;
})
},
getRatings() {
return apiClient.get('/api/analysis/by-rating')
.then(response => {
return response.data.data;
})
},
getYears() {
return apiClient.get('/api/analysis/by-year')
.then(response => {
return response.data.data;
})
},
getCountries() {
return apiClient.get('/api/analysis/by-country')
.then(response => {
return response.data.data;
})
},
getLanguages() {
return apiClient.get('/api/analysis/by-language')
.then(response => {
return response.data.data;
})
},
getDurations() {
return apiClient.get('/api/analysis/by-duration')
.then(response => {
return response.data.data;
})
},
getDirectors() {
return apiClient.get('/api/analysis/by-director')
.then(response => {
return response.data.data;
})
},
getContentRatings() {
return apiClient.get('/api/analysis/by-content-rating')
.then(response => {
return response.data.data;
})
},
getAwards() {
return apiClient.get('/api/analysis/by-awards')
.then(response => {
return response.data.data;
})
},
getCriticReviews() {
return apiClient.get('/api/analysis/by-critic-reviews')
.then(response => {
return response.data.data;
})
}
}

4.3 图表组件实现

4.3.1 类型分布饼图

<template>
<div class="chart-genre">
<div v-if="emptyData" class="no-data">暂无类型数据</div>
<div v-else class="chart-container" ref="chart"></div>
</div>
</template>

<script>
import { ref, watch, onMounted } from 'vue'
import * as echarts from 'echarts'

export default {
name: 'ChartGenre',
props: {
chartData: {
type: [Array, Object],
required: true
}
},
setup(props) {
const chart = ref(null)
const chartInstance = ref(null)
const emptyData = ref(false)
const chartOption = ref({})

const baseChartOptions = {
title: {
text: '',
left: 'center',
textStyle: {
fontSize: 16,
fontWeight: 'bold'
}
},
tooltip: {
trigger: 'item'
},
legend: {
orient: 'vertical',
right: 10,
top: 'center'
}
}

const initChart = () => {
if (!chart.value) return

chartInstance.value = echarts.init(chart.value)
window.addEventListener('resize', resizeChart)
}

const resizeChart = () => {
if (chartInstance.value) {
chartInstance.value.resize()
}
}

const processData = () => {
try {
let genreData = []

if (Array.isArray(props.chartData)) {
props.chartData.forEach(item => {
const name = item.genre || item.type || item.name || item.label || '未知'
const value = item.value || item.count || item.total || 0

genreData.push({
name,
value: Number(value)
})
})
}
else if (typeof props.chartData === 'object' && props.chartData !== null) {
Object.entries(props.chartData).forEach(([name, value]) => {
genreData.push({
name,
value: Number(value)
})
})
}

genreData = genreData.filter(item => item.value > 0)

if (genreData.length === 0) {
emptyData.value = true
return
}

genreData.sort((a, b) => b.value – a.value)

const total = genreData.reduce((sum, item) => sum + item.value, 0)

chartOption.value = {
…baseChartOptions,
title: {
…baseChartOptions.title,
text: `电影类型分布 (共 ${total} 部)`
},
series: [
{
name: '电影类型',
type: 'pie',
radius: ['40%', '70%'],
avoidLabelOverlap: false,
itemStyle: {
borderRadius: 10,
borderColor: '#fff',
borderWidth: 2
},
label: {
show: true,
formatter: '{b}: {c} ({d}%)'
},
emphasis: {
label: {
show: true,
fontSize: '18',
fontWeight: 'bold'
}
},
labelLine: {
show: true
},
data: genreData
}
]
}

emptyData.value = false
if (chartInstance.value) {
updateChart()
}
} catch (error) {
console.error('Error processing chart data:', error)
emptyData.value = true
}
}

const updateChart = () => {
if (chartInstance.value && chartOption.value) {
chartInstance.value.setOption(chartOption.value)
}
}

onMounted(() => {
initChart()
processData()
})

watch(chart, (newVal) => {
if (newVal && !chartInstance.value) {
initChart()
if (Object.keys(chartOption.value).length > 0) {
updateChart()
}
}
})

watch(() => props.chartData, () => {
processData()
}, { deep: true, immediate: true })

return {
chart,
emptyData
}
}
}
</script>

4.3.2 评分分布柱状图

<template>
<div class="chart-rating">
<div v-if="emptyData" class="no-data">暂无评分数据</div>
<div v-else class="chart-container" ref="chart"></div>
</div>
</template>

<script>
import { ref, watch, onMounted } from 'vue'
import * as echarts from 'echarts'

export default {
name: 'ChartRating',
props: {
chartData: {
type: [Array, Object],
required: true
}
},
setup(props) {
const chart = ref(null)
const chartInstance = ref(null)
const emptyData = ref(false)
const chartOption = ref({})

const initChart = () => {
if (!chart.value) return
chartInstance.value = echarts.init(chart.value)
window.addEventListener('resize', resizeChart)
}

const resizeChart = () => {
if (chartInstance.value) {
chartInstance.value.resize()
}
}

const processData = () => {
try {
let ratingData = []

if (Array.isArray(props.chartData)) {
ratingData = props.chartData.map(item => ({
name: item.name || item.label || item.range || '未知',
value: item.value || item.count || 0
}))
} else if (typeof props.chartData === 'object' && props.chartData !== null) {
ratingData = Object.entries(props.chartData).map(([name, value]) => ({
name,
value: Number(value)
}))
}

ratingData = ratingData.filter(item => item.value > 0)

if (ratingData.length === 0) {
emptyData.value = true
return
}

const xAxisData = ratingData.map(item => item.name)
const seriesData = ratingData.map(item => item.value)

chartOption.value = {
title: {
text: '电影评分分布',
left: 'center',
textStyle: {
fontSize: 16,
fontWeight: 'bold'
}
},
tooltip: {
trigger: 'axis',
axisPointer: {
type: 'shadow'
}
},
grid: {
left: '3%',
right: '4%',
bottom: '3%',
containLabel: true
},
xAxis: {
type: 'category',
data: xAxisData,
axisLabel: {
rotate: 0
}
},
yAxis: {
type: 'value'
},
series: [
{
name: '电影数量',
type: 'bar',
data: seriesData,
itemStyle: {
color: {
type: 'linear',
x: 0,
y: 0,
x2: 0,
y2: 1,
colorStops: [
{ offset: 0, color: '#83bff6' },
{ offset: 0.5, color: '#188df0' },
{ offset: 1, color: '#188df0' }
]
}
},
emphasis: {
itemStyle: {
color: {
type: 'linear',
x: 0,
y: 0,
x2: 0,
y2: 1,
colorStops: [
{ offset: 0, color: '#2378f7' },
{ offset: 0.7, color: '#2378f7' },
{ offset: 1, color: '#83bff6' }
]
}
}
}
}
]
}

emptyData.value = false
if (chartInstance.value) {
updateChart()
}
} catch (error) {
console.error('Error processing chart data:', error)
emptyData.value = true
}
}

const updateChart = () => {
if (chartInstance.value && chartOption.value) {
chartInstance.value.setOption(chartOption.value)
}
}

onMounted(() => {
initChart()
processData()
})

watch(chart, (newVal) => {
if (newVal && !chartInstance.value) {
initChart()
if (Object.keys(chartOption.value).length > 0) {
updateChart()
}
}
})

watch(() => props.chartData, () => {
processData()
}, { deep: true, immediate: true })

return {
chart,
emptyData
}
}
}
</script>

4.4 看板页面

<template>
<div class="dashboard-view">
<div class="dashboard-header">
<h1>电影数据分析看板</h1>
<div class="dashboard-controls">
<el-select v-model="yearFilter" placeholder="选择年份" clearable>
<el-option
v-for="year in availableYears"
:key="year"
:label="year"
:value="year"
/>
</el-select>
<el-select v-model="genreFilter" placeholder="选择类型" clearable>
<el-option
v-for="genre in availableGenres"
:key="genre"
:label="genre"
:value="genre"
/>
</el-select>
</div>
</div>

<div class="dashboard-content">
<div class="chart-row">
<el-card class="chart-card">
<ChartRating :chartData="filteredRatingData" />
</el-card>
<el-card class="chart-card">
<ChartGenre :chartData="filteredGenreData" />
</el-card>
</div>
<div class="chart-row">
<el-card class="chart-card">
<ChartRevenue :chartData="filteredRevenueData" />
</el-card>
<el-card class="chart-card">
<ChartWordCloud />
</el-card>
</div>
</div>
</div>
</template>

<script>
import { ref, computed } from 'vue'
import ChartRating from '@/components/ChartRating.vue'
import ChartGenre from '@/components/ChartGenre.vue'
import ChartRevenue from '@/components/ChartRevenue.vue'
import ChartWordCloud from '@/components/ChartWordCloud.vue'

export default {
name: 'DashboardView',
components: {
ChartRating,
ChartGenre,
ChartRevenue,
ChartWordCloud
},
setup() {
const ratingData = ref([
{ name: '<6', value: 16644 },
{ name: '6-7', value: 8662 },
{ name: '7-8', value: 8433 },
{ name: '8-9', value: 8458 },
{ name: '9+', value: 7803 }
])

const genreData = ref([
{ name: '动作', value: 12000 },
{ name: '喜剧', value: 9800 },
{ name: '爱情', value: 7500 },
{ name: '科幻', value: 6800 },
{ name: '恐怖', value: 5200 }
])

const revenueData = ref([
{ year: '2018', revenue: 56.8 },
{ year: '2019', revenue: 62.3 },
{ year: '2020', revenue: 30.4 },
{ year: '2021', revenue: 47.2 },
{ year: '2022', revenue: 60.0 }
])

const yearFilter = ref('')
const genreFilter = ref('')

const availableYears = computed(() => {
return […new Set(revenueData.value.map(item => item.year))]
})

const availableGenres = computed(() => {
return […new Set(genreData.value.map(item => item.name))]
})

const filteredRatingData = computed(() => {
return ratingData.value
})

const filteredGenreData = computed(() => {
return genreData.value
})

const filteredRevenueData = computed(() => {
if (!yearFilter.value) return revenueData.value
return revenueData.value.filter(item => item.year === yearFilter.value)
})

return {
yearFilter,
genreFilter,
availableYears,
availableGenres,
filteredRatingData,
filteredGenreData,
filteredRevenueData
}
}
}
</script>

<style scoped>
.dashboard-view {
padding: 20px;
max-width: 1400px;
margin: 0 auto;
}

.dashboard-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 20px;
}

.dashboard-header h1 {
margin: 0;
font-size: 24px;
color: #333;
}

.dashboard-controls {
display: flex;
gap: 10px;
}

.dashboard-content {
display: flex;
flex-direction: column;
gap: 20px;
}

.chart-row {
display: flex;
gap: 20px;
}

.chart-card {
flex: 1;
min-height: 400px;
}

@media (max-width: 768px) {
.chart-row {
flex-direction: column;
}

.dashboard-header {
flex-direction: column;
align-items: flex-start;
gap: 10px;
}
}
</style>

五、核心功能详解

5.1 多维度数据分析

系统支持从12个维度对电影数据进行分析：

5.2 数据可视化

系统使用ECharts实现了多种图表类型：

• 饼图：展示类型、国家、语言等分类数据的占比
• 柱状图：展示年份、评分、时长等数据的分布
• 折线图：展示票房等数据的趋势变化
• 散点图：展示预算与票房的关系
• 词云图：展示电影标题的关键词

5.3 交互式筛选

看板支持按年份和类型进行数据筛选，用户可以：

• 选择特定年份查看该年的数据
• 选择特定类型查看该类型的统计信息
• 组合多个筛选条件进行精确查询

六、技术亮点

6.1 大数据处理

• Hive数据仓库：使用Hive构建数据仓库，支持大规模数据存储和查询
• Spark批处理：利用Spark进行高效的数据清洗和分析
• Sqoop数据迁移：实现Hadoop与关系型数据库之间的数据传输

6.2 后端架构

• Spring Boot：快速构建RESTful API
• MyBatis Plus：简化数据库操作，提供CRUD接口
• Java 8 Stream API：函数式编程风格，代码简洁高效
• 统一响应封装：ApiResponse统一处理API响应格式

6.3 前端技术

• Vue 3组合式API：更好的代码组织和复用性
• ECharts可视化：丰富的图表类型和交互功能
• Vite构建工具：快速的开发体验和构建速度
• Axios HTTP客户端：统一的请求拦截和错误处理
• 响应式设计：支持多种屏幕尺寸

6.4 性能优化

• 数据库索引：为常用查询字段创建索引
• 数据缓存：前端使用计算属性缓存计算结果
• 懒加载：图表组件按需加载
• 分页查询：支持大数据量的分页展示

七、部署与运行

7.1 环境要求

• JDK 1.8+
• Node.js 16+
• MySQL 8.0+
• Hadoop 3.x
• Hive 3.x
• Spark 3.x
• Sqoop 1.4.x

7.2 启动步骤

7.2.1 启动Hadoop集群

start-dfs.sh
start-yarn.sh

7.2.2 启动Hive

hive –service metastore &
hive –service hiveserver2 &

7.2.3 执行Spark分析

spark-submit spark_movie_analysis.py

7.2.4 导出数据到MySQL

bash import_to_mysql.sh

7.2.5 启动后端服务

cd movie-analysis-api
mvn spring-boot:run

7.2.6 启动前端服务

cd movie-analysis-dashboard
npm install
npm run dev

7.3 访问地址

• 前端看板：http://localhost:5173
• 后端API：http://localhost:8080
• API文档：http://localhost:8080/swagger-ui.html

八、总结

本项目实现了一个完整的电影数据分析系统，涵盖了从数据采集、处理、分析到可视化展示的全流程。主要特点包括：

该系统为电影行业从业者提供了强大的数据分析工具，可以帮助他们更好地了解市场趋势、观众偏好和投资回报，为决策提供数据支持。

九、源码地址

完整源码已上传至GitHub，欢迎Star和Fork：

项目地址：https://m.tb.cn/h.7upHdAa?tk=Mp07UkIas74 CZ001

作者：大数据基础 日期：2026-02-08 标签：大数据、Spring Boot、Vue、ECharts、数据分析